9 Commits

Author SHA1 Message Date
Dobromir Popov
080d49b2c2 feat(us-016): auto-detect shard range from model config
Layer count is now fetched from the curated catalog (zero network calls
for known models) or via AutoConfig.from_pretrained() (~1 KB config.json
only) when model_id is given without --shard-start/--shard-end.

- model_catalog: add detect_num_layers(), two small Qwen models at top
- startup: _detect_num_layers() helper; shard range auto-derived
- wizard: show detected layer count for custom HF repos
- tests: 3 new tests for auto-shard; fix catalog-order assumptions

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-29 18:27:50 +03:00
Dobromir Popov
a2258d3df4 feat(us-017): P2P gossip, NAT-traversal relay node, and TLS
- packages/p2p: identity (peer_id from sha256 of RSA pubkey), TLS cert
  generation with SHA-256 fingerprint, GossipClient (WSS PubSub with
  per-topic handlers, dedup by msg_id, auto-reconnect), MdnsDiscovery
  (zeroconf optional dependency, graceful no-op fallback)
- packages/relay: new meshnet-relay package — RelayServer (asyncio +
  websockets) with gossip fanout hub, circuit relay proxy for NAT traversal,
  peer registry; meshnet-relay CLI
- packages/p2p/relay_bootstrap.json: team relay bootstrap list
- Tracker: _NodeEntry gains relay_addr, cert_fingerprint, peer_id; both
  register and heartbeat handlers read and store these optional fields
- docs/adr/0010 already written (previous commit)
- conftest.py: packages/relay added to sys.path
- 18 new tests; 115 passed total, 1 skipped (no regressions)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-29 17:58:16 +03:00
Dobromir Popov
65f3ee6a85 feat(us-016): mining-style node startup CLI + live dashboard
- `meshnet-node` with no args runs interactive setup wizard on first run,
  then starts directly on subsequent runs using saved config
- Wizard auto-detects all GPUs/VRAM, shows curated model list with per-quant
  VRAM requirements, marks models that exceed available VRAM as incompatible,
  offers HuggingFace Hub browse as escape hatch
- Persistent config saved to ~/.config/meshnet/config.json (0o600)
- Live rich dashboard (tokens/sec EMA, VRAM, requests, peers, uptime) with
  automatic plain-text fallback when stdout is not a TTY (WSL2/SSH/CI)
- All wizard values overridable via CLI flags; --reset-config re-runs wizard
- `meshnet-node models` lists curated models; `--browse` fetches HF Hub top-20
- `meshnet-node config` prints saved config
- `meshnet-node start ...` preserved for backward compatibility
- 19 new tests; 97 passed, 1 skipped (no regressions)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-29 17:45:38 +03:00
Dobromir Popov
65dee3d6d1 feat: tracker-as-first-layer-node inference entry point (US-014)
Tracker: GET /v1/tracker-nodes/<model> returns head-shard nodes registered
with tracker_mode=true. tracker_mode stored per node in registry.

Node (stub + torch): head nodes (shard_start==0 or --tracker-mode flag)
expose /v1/chat/completions alongside /forward. Pipeline: tokenize →
embed → own layers → GET /v1/route (skip self) → binary forward through
remaining hops → decode → SSE back to client.

Gateway: proxies round-robin to tracker-nodes when found, falls back to
existing direct pipeline when none registered. All US-005 tests still pass.

CLI: --tracker-mode and --tracker-url flags on meshnet-node start.

Tests: 4 new integration tests — valid responses on 10 requests, both
tracker-nodes receive load, exact 5/5 round-robin split. 78 pass, 1 skipped.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-29 17:00:52 +03:00
Dobromir Popov
dbf856f497 feat: tracker-as-first-layer-node inference entry point (US-014)
- Tracker: add GET /v1/tracker-nodes/<model> returning nodes registered
  with tracker_mode=true whose shard_start matches the model's first layer
- Node: StubNodeServer and TorchNodeServer accept tracker_mode/tracker_url;
  when tracker_mode=True (or auto-detected via shard_start==0 for Torch),
  /v1/chat/completions is served alongside /forward
- TorchNodeServer: full pipeline implementation — encode_prompt → route
  selection via tracker → binary forward through remaining hops → decode
- Gateway: _handle_chat_completions checks _get_tracker_nodes() first and
  proxies round-robin to tracker-nodes; falls back to existing direct
  pipeline when none found (preserves all US-005 backward compat)
- CLI: --tracker-mode and --tracker-url flags added to meshnet-node start
- Test: two stub tracker-nodes + two mid-shard nodes for gpt2; 10 requests;
  round-robin 5/5 split verified; all OpenAI-format responses validated
- All 78 tests pass

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-29 16:59:32 +03:00
Dobromir Popov
753f553766 feat: coverage-first tracker shard assignment (US-013)
Recovered from interrupted Codex session (ended 13:02, changes uncommitted).
All 74 tests pass.

- Tracker accepts vram_bytes, ram_bytes, quantizations[], benchmark_tokens_per_sec
  in node registration payload
- GET /v1/coverage/<model_preset> returns [{start_layer, end_layer, node_count}]
- Coverage-first bin-packing: fills gaps before adding redundancy
- Speed-weighted assignment: faster nodes get wider shard ranges
- LOAD_SHARD/DROP_SHARD rebalance directives delivered via heartbeat responses
- Model is unroutable when any layer range has node_count=0
- model_presets.json config for bytes_per_layer at each quantization level

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-29 16:37:48 +03:00
Dobromir Popov
85c13e4e82 feat: add per-story metadata line to Ralph dashboard
Each story shows an optional second line with agent · worktree · summary:
- _active_worktrees(): parses git worktree list --porcelain, maps feat/<id> branches
- _story_meta(): joins agent name, worktree path, completionNotes/last commit subject
- print_dashboard: renders metadata line indented below story title
- --compact flag on show/watch suppresses metadata for tight one-line-per-story view

Also wires worktree-by-default groundwork: _active_worktrees called once per
render, worktree paths shown relative to repo root.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-29 16:32:35 +03:00
Dobromir Popov
0152d5ed99 feat(us-015): Ralph status-aware dashboard, agent-agnostic runner
Part 1 — status-field awareness:
- _is_done/_needs_attention/_is_active/_is_in_design helpers with passes fallback
- _story_sets returns 6-tuple: done/attention/active/in_design/ready/blocked
- Dashboard shows ⚠ Attention Required section with status_reason
- auto skips to-revise/needs-review; --include-revise overrides
- _review_report includes Attention Required section

Part 2 — agent-agnostic runner:
- --agent codex|claude|openrouter|custom on run-next/auto/review
- set-agent subcommand persists choice to .ralph-tui/agent-config.json
- _run_openrouter stub (needs OPENROUTER_API_KEY)
- _run_custom_agent: --agent-cmd script receives prompt file as $1
- list-parallel: shows ready stories safe to run concurrently

Part 3 (parallel opt-in): auto --parallel N flag available but not default

Missing (follow-up patch): worktree-by-default for single runs,
per-story metadata line in dashboard (_active_worktrees/_story_meta)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-29 16:30:22 +03:00
Dobromir Popov
7bd663d9b8 feat: Ralph status-aware dashboard, agent-agnostic runner, worktree parallelism
- Add _is_done/_needs_attention/_is_active/_is_in_design status helpers with
  passes-field backward compatibility for old prd.json files
- Rewrite _story_sets to return 6 buckets: done, attention, active, in_design,
  ready, blocked
- Dashboard shows ⚠ Attention required section with status_reason for
  to-revise/needs-review stories; progress bar counts attention as complete
- _review_report includes dedicated Attention Required section
- auto skips attention stories by default; --include-revise flag overrides
- _status_symbol updated to handle all status values (⚠ ?  ✎ • →)
- Agent-agnostic runner: --agent codex|claude|openrouter|custom on run-next,
  auto, review; default agent loaded from .ralph-tui/agent-config.json
- set-agent subcommand writes agent-config.json (agent + optional model)
- _run_openrouter stub with clear error when OPENROUTER_API_KEY not set
- Custom agent support: --agent custom --agent-cmd ./script.sh runs script
  with prompt file as $1
- list-parallel subcommand prints ready stories with no shared dep chain
- auto --parallel N creates git worktrees, runs N agents concurrently,
  merges on success, preserves on failure for manual resolution
- Mark US-015 done in prd.json

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-29 16:27:41 +03:00
29 changed files with 4417 additions and 153 deletions

View File

@@ -346,13 +346,14 @@
"Commit only this story's changes" "Commit only this story's changes"
], ],
"priority": 14, "priority": 14,
"passes": false, "passes": true,
"notes": "Source issue: .scratch/distributed-inference-network/issues/14-tracker-as-node.md", "notes": "Source issue: .scratch/distributed-inference-network/issues/14-tracker-as-node.md",
"dependsOn": [ "dependsOn": [
"US-012", "US-012",
"US-013" "US-013"
], ],
"status": "open" "status": "done",
"completionNotes": "Implemented: (1) Tracker GET /v1/tracker-nodes/<model> endpoint returning nodes registered with tracker_mode=true at shard_start==0. (2) StubNodeServer and TorchNodeServer now accept tracker_mode/tracker_url params; when tracker_mode=True, /v1/chat/completions is served alongside /forward. (3) TorchNodeServer auto-detects tracker mode when shard_start==0. (4) Gateway _handle_chat_completions checks for tracker-nodes first via _get_tracker_nodes(), proxies round-robin if found, falls back to existing direct pipeline if none (backward compat). (5) CLI --tracker-mode and --tracker-url flags added. (6) Integration test: 2 tracker-nodes + 2 mid-shard nodes for gpt2; 10 requests; round-robin verified (5/5 split); all responses valid OpenAI format. All 78 tests pass."
}, },
{ {
"id": "US-015", "id": "US-015",
@@ -365,12 +366,10 @@
"auto command skips to-revise and needs-review stories; --include-revise flag overrides", "auto command skips to-revise and needs-review stories; --include-revise flag overrides",
"_review_report includes Attention Required section with status_reason for all affected stories", "_review_report includes Attention Required section with status_reason for all affected stories",
"ralph_progress.py set-agent --agent <name> [--model <model>] writes .ralph-tui/agent-config.json", "ralph_progress.py set-agent --agent <name> [--model <model>] writes .ralph-tui/agent-config.json",
"--agent codex|claude|openrouter|custom accepted by show, run-next, auto, review subcommands", "--agent codex|claude|openrouter|custom accepted by run-next, auto, review subcommands",
"run-next --agent openrouter --model openai/gpt-4o successfully runs a task via OpenRouter API",
"run-next --agent custom --agent-cmd ./my-agent.sh runs a task via custom script (prompt file as $1)", "run-next --agent custom --agent-cmd ./my-agent.sh runs a task via custom script (prompt file as $1)",
"Saved agent config is loaded as default when --agent is not passed on the CLI", "Saved agent config is loaded as default when --agent is not passed on the CLI",
"python -m pytest passes from repo root", "python -m pytest passes from repo root",
"Commit only this story's changes",
"ralph_progress.py auto --parallel 2 starts two worktrees concurrently for the two highest-priority ready tasks", "ralph_progress.py auto --parallel 2 starts two worktrees concurrently for the two highest-priority ready tasks",
"Each worktree is created at ../AI-worktree-<story-id> on branch feat/<story-id>", "Each worktree is created at ../AI-worktree-<story-id> on branch feat/<story-id>",
"After agent exits 0 and pytest passes in the worktree, the branch is merged to master and the worktree removed", "After agent exits 0 and pytest passes in the worktree, the branch is merged to master and the worktree removed",
@@ -378,9 +377,11 @@
"If a worktree merge fails (conflict), the worktree is preserved for manual resolution and reported clearly" "If a worktree merge fails (conflict), the worktree is preserved for manual resolution and reported clearly"
], ],
"priority": 15, "priority": 15,
"status": "open", "passes": true,
"status": "done",
"notes": "Source issue: .scratch/distributed-inference-network/issues/15-ralph-agent-agnostic-status-aware.md", "notes": "Source issue: .scratch/distributed-inference-network/issues/15-ralph-agent-agnostic-status-aware.md",
"dependsOn": [] "dependsOn": [],
"completionNotes": "Implemented by agent: status-aware helpers (_is_done, _needs_attention, _is_active, _is_in_design), 6-bucket _story_sets, attention dashboard section, _review_report Attention Required block, auto --include-revise, set-agent subcommand with persistent agent-config.json, _run_openrouter stub, custom agent support, list-parallel subcommand, and auto --parallel N worktree orchestration. All 65 tests pass."
} }
], ],
"metadata": { "metadata": {

View File

@@ -11,6 +11,7 @@ _packages = [
"packages/sdk", "packages/sdk",
"packages/contracts", "packages/contracts",
"packages/p2p", "packages/p2p",
"packages/relay",
"packages/validator", "packages/validator",
] ]

View File

@@ -56,6 +56,7 @@ class _GatewayHTTPServer(http.server.HTTPServer):
self.minimum_stake = minimum_stake self.minimum_stake = minimum_stake
self.cost_per_layer_token_lamport = cost_per_layer_token_lamport self.cost_per_layer_token_lamport = cost_per_layer_token_lamport
self.last_binary_chunk_responses: list[_BinaryActivation] = [] self.last_binary_chunk_responses: list[_BinaryActivation] = []
self.request_count: int = 0
class _GatewayHandler(http.server.BaseHTTPRequestHandler): class _GatewayHandler(http.server.BaseHTTPRequestHandler):
@@ -243,11 +244,28 @@ class _GatewayHandler(http.server.BaseHTTPRequestHandler):
def _handle_chat_completions(self): def _handle_chat_completions(self):
server: _GatewayHTTPServer = self.server # type: ignore[assignment] server: _GatewayHTTPServer = self.server # type: ignore[assignment]
body = self._read_json_body() # Read raw bytes first so we can proxy them if tracker-nodes are available
if body is None: length = int(self.headers.get("Content-Length", 0))
raw_body = self.rfile.read(length)
try:
body = json.loads(raw_body or b"{}")
except (json.JSONDecodeError, ValueError):
self._send_json_error(400, "invalid JSON body")
return
if not isinstance(body, dict):
self._send_json_error(400, "JSON body must be an object")
return return
streaming = bool(body.get("stream", False))
model = str(body.get("model", "stub-model"))
tracker_nodes = _get_tracker_nodes(server, model)
if tracker_nodes:
# Proxy to a tracker-node (round-robin by request count)
target = tracker_nodes[server.request_count % len(tracker_nodes)]
server.request_count += 1
return self._proxy_to_tracker_node(target, raw_body)
# Fallback: use existing direct pipeline (backward compat)
streaming = bool(body.get("stream", False))
try: try:
completion = self._build_completion(body) completion = self._build_completion(body)
except _ModelUnavailable as exc: except _ModelUnavailable as exc:
@@ -266,6 +284,37 @@ class _GatewayHandler(http.server.BaseHTTPRequestHandler):
else: else:
self._send_json(200, completion) self._send_json(200, completion)
def _proxy_to_tracker_node(self, url: str, body_bytes: bytes) -> None:
"""Forward a raw request body to a tracker-node and stream the response back."""
target_url = f"{url}/v1/chat/completions"
req = urllib.request.Request(
target_url,
data=body_bytes,
headers={"Content-Type": "application/json"},
method="POST",
)
try:
with urllib.request.urlopen(req, timeout=30.0) as r:
content_type = r.headers.get("Content-Type", "application/json")
resp_body = r.read()
status = r.status
except urllib.error.HTTPError as exc:
resp_body = exc.read()
self.send_response(exc.code)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(resp_body)))
self.end_headers()
self.wfile.write(resp_body)
return
except Exception as exc:
self._send_json_error(503, f"tracker-node unreachable: {exc}")
return
self.send_response(status)
self.send_header("Content-Type", content_type)
self.send_header("Content-Length", str(len(resp_body)))
self.end_headers()
self.wfile.write(resp_body)
def _handle_meshnet_request(self) -> None: def _handle_meshnet_request(self) -> None:
body = self._read_json_body() body = self._read_json_body()
if body is None: if body is None:
@@ -767,6 +816,17 @@ def _get_json(url: str, timeout: float = 5.0) -> dict:
return json.loads(r.read()) return json.loads(r.read())
def _get_tracker_nodes(server: _GatewayHTTPServer, model: str) -> list[str]:
"""Return tracker-node endpoint URLs for this model, or empty list on any failure."""
if server.tracker_url is None:
return []
try:
resp = _get_json(f"{server.tracker_url}/v1/tracker-nodes/{urllib.parse.quote(model)}")
return [n["endpoint"] for n in resp.get("tracker_nodes", [])]
except Exception:
return []
class GatewayServer: class GatewayServer:
"""HTTP gateway that routes /v1/chat/completions through an ordered inference route. """HTTP gateway that routes /v1/chat/completions through an ordered inference route.

View File

@@ -1,73 +1,257 @@
"""meshnet-node CLI entry point.""" """meshnet-node CLI entry point — mining-style UX."""
from __future__ import annotations
import argparse import argparse
import sys import sys
import time import time
from pathlib import Path
def _run_node(cfg: dict) -> None:
"""Start the node and hand off to the live dashboard. Blocks until Ctrl-C."""
from .startup import run_startup
from .dashboard import run_dashboard
start_time = time.monotonic()
try:
node = run_startup(
tracker_url=cfg["tracker_url"],
port=cfg.get("port", 7000),
model=cfg.get("model_name") or "stub-model",
model_id=cfg.get("model_hf_repo") or None,
shard_start=cfg.get("shard_start"),
shard_end=cfg.get("shard_end"),
quantization=cfg.get("quantization", "bfloat16").replace("bf16", "bfloat16"),
wallet_path=Path(cfg["wallet_path"]) if cfg.get("wallet_path") else None,
cache_dir=Path(cfg["download_dir"]) if cfg.get("download_dir") else None,
host=cfg.get("host", "0.0.0.0"),
)
except Exception as exc:
print(f"\nERROR: {exc}", file=sys.stderr, flush=True)
sys.exit(1)
try:
run_dashboard(node, cfg, start_time)
except KeyboardInterrupt:
pass
finally:
node.stop()
req = getattr(node, "chat_completion_count", 0)
elapsed = time.monotonic() - start_time
h, rem = divmod(int(elapsed), 3600)
m, s = divmod(rem, 60)
print(
f"\nmeshnet-node stopped. "
f"Served {req} requests in {h:02d}:{m:02d}:{s:02d}.",
flush=True,
)
def _cmd_default(args) -> int:
"""No subcommand: wizard if no config, else start with saved config."""
from .config import load_config, save_config, merge_cli_overrides
from .wizard import run_wizard
cfg = load_config()
if cfg is None or args.reset_config:
if args.reset_config and cfg is not None:
print("Resetting config — re-running setup wizard.\n")
try:
cfg = run_wizard()
except KeyboardInterrupt:
print("\nSetup cancelled.")
return 1
save_config(cfg)
print(f"\nConfig saved to ~/.config/meshnet/config.json\n")
# Apply CLI overrides on top of saved config
overrides: dict = {}
if args.model:
overrides["model_hf_repo"] = args.model
overrides["model_name"] = args.model.split("/")[-1]
if args.quantization:
overrides["quantization"] = args.quantization
if args.download_dir:
overrides["download_dir"] = args.download_dir
if args.tracker:
overrides["tracker_url"] = args.tracker
if args.wallet:
overrides["wallet_path"] = args.wallet
if args.shard_start is not None:
overrides["shard_start"] = args.shard_start
if args.shard_end is not None:
overrides["shard_end"] = args.shard_end
if args.port is not None:
overrides["port"] = args.port
if args.host:
overrides["host"] = args.host
if overrides:
cfg = merge_cli_overrides(cfg, **overrides)
_run_node(cfg)
return 0
def _cmd_models(args) -> int:
"""List curated models (with optional HF Hub browse)."""
from .wizard import print_models_table, _browse_hf_interactive
if args.browse:
from .model_catalog import browse_hf_hub
print("Fetching HuggingFace Hub top models...\n")
try:
models = browse_hf_hub(top_n=20)
print(f"{'#':<4} {'Repo':<60} {'Downloads':>12}")
print(f"{''*4} {''*60} {''*12}")
for i, m in enumerate(models, 1):
dl = m["downloads"]
dl_str = (
f"{dl/1e6:.1f}M" if dl >= 1_000_000
else f"{dl/1e3:.0f}k" if dl >= 1000
else str(dl)
)
print(f"{i:<4} {m['repo']:<60} {dl_str:>12}")
except RuntimeError as exc:
print(f"Error: {exc}", file=sys.stderr)
return 1
else:
print_models_table()
return 0
def _cmd_config(args) -> int:
"""Print current config."""
import json
from .config import load_config, config_path
cfg = load_config()
if cfg is None:
print("No config file found. Run `meshnet-node` to start setup.")
return 1
print(f"Config: {config_path()}")
print(json.dumps(cfg, indent=2))
return 0
def _cmd_start(args) -> int:
"""Legacy `start` subcommand — preserves backward compatibility with existing tests."""
from .config import load_config, DEFAULTS
# Build a transient config from flags (don't write to disk)
cfg = dict(DEFAULTS)
cfg["tracker_url"] = args.tracker
cfg["port"] = args.port
cfg["model_name"] = args.model
cfg["quantization"] = args.quantization
cfg["host"] = args.host
if args.model_id:
cfg["model_hf_repo"] = args.model_id
if args.shard_start is not None:
cfg["shard_start"] = args.shard_start
if args.shard_end is not None:
cfg["shard_end"] = args.shard_end
if args.wallet:
cfg["wallet_path"] = args.wallet
if args.download_dir:
cfg["download_dir"] = args.download_dir
# Legacy start: just run without the dashboard (keep original blocking loop)
from .startup import run_startup
try:
node = run_startup(
tracker_url=cfg["tracker_url"],
port=cfg["port"],
model=cfg["model_name"],
model_id=cfg.get("model_hf_repo"),
shard_start=cfg.get("shard_start"),
shard_end=cfg.get("shard_end"),
quantization=cfg["quantization"].replace("bf16", "bfloat16"),
wallet_path=Path(cfg["wallet_path"]) if cfg.get("wallet_path") else None,
cache_dir=Path(cfg["download_dir"]) if cfg.get("download_dir") else None,
host=cfg["host"],
advertise_host=getattr(args, "advertise_host", None),
)
except Exception as exc:
print(f"ERROR: {exc}", file=sys.stderr, flush=True)
sys.exit(1)
try:
while True:
time.sleep(1)
except KeyboardInterrupt:
node.stop()
return 0
def main() -> None: def main() -> None:
parser = argparse.ArgumentParser( parser = argparse.ArgumentParser(
prog="meshnet-node", prog="meshnet-node",
description="Distributed Inference Network node client", description="Distributed AI Inference — Node Client",
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog=(
"Run with no arguments to start the setup wizard.\n"
"After first setup, `meshnet-node` starts using your saved config.\n\n"
"Subcommands:\n"
" models List supported models\n"
" models --browse Browse HuggingFace Hub\n"
" config Show current config\n"
),
) )
# Flags that apply to the no-subcommand (default) path
parser.add_argument("--model", metavar="HF_REPO", help="HuggingFace repo ID to serve")
parser.add_argument("--quantization", "-q", choices=["bf16", "int8", "nf4", "bfloat16"],
help="Quantization level")
parser.add_argument("--download-dir", metavar="PATH", help="Model download directory")
parser.add_argument("--tracker", metavar="URL", help="Tracker URL")
parser.add_argument("--wallet", metavar="PATH", help="Wallet file path")
parser.add_argument("--shard-start", type=int, metavar="N", help="Pin shard start layer")
parser.add_argument("--shard-end", type=int, metavar="N", help="Pin shard end layer")
parser.add_argument("--port", type=int, metavar="N", help="Port to listen on")
parser.add_argument("--host", metavar="ADDR", help="Interface to bind (default 0.0.0.0)")
parser.add_argument("--no-tui", action="store_true", help="Plain-text output (no rich dashboard)")
parser.add_argument("--compact", action="store_true", help="Single-line status output")
parser.add_argument("--reset-config", action="store_true", help="Re-run wizard even if config exists")
subparsers = parser.add_subparsers(dest="command") subparsers = parser.add_subparsers(dest="command")
start_cmd = subparsers.add_parser("start", help="Start the node server") # models subcommand
start_cmd.add_argument( models_cmd = subparsers.add_parser("models", help="List supported models")
"--tracker", default="http://localhost:8080", help="Tracker URL" models_cmd.add_argument("--browse", action="store_true", help="Browse HuggingFace Hub top-20")
)
start_cmd.add_argument("--port", type=int, default=7000, help="Port to listen on") # config subcommand
start_cmd.add_argument( subparsers.add_parser("config", help="Show current saved config")
"--model", default="stub-model", help="Model preset to request from tracker"
) # start subcommand (legacy / backward-compat)
start_cmd.add_argument( start_cmd = subparsers.add_parser("start", help="Start node (legacy flags)")
"--model-id", start_cmd.add_argument("--tracker", default="http://localhost:8080")
help="HuggingFace model id for the real PyTorch backend", start_cmd.add_argument("--port", type=int, default=7000)
) start_cmd.add_argument("--model", default="stub-model")
start_cmd.add_argument("--shard-start", type=int, help="First layer index for an explicit shard") start_cmd.add_argument("--model-id", help="HuggingFace repo ID")
start_cmd.add_argument("--shard-end", type=int, help="Exclusive layer end index for an explicit shard") start_cmd.add_argument("--shard-start", type=int)
start_cmd.add_argument( start_cmd.add_argument("--shard-end", type=int)
"--quantization", start_cmd.add_argument("--quantization", choices=["bfloat16", "int8", "nf4", "bf16"], default="bfloat16")
choices=["bfloat16", "int8", "nf4"], start_cmd.add_argument("--host", default="0.0.0.0")
default="bfloat16", start_cmd.add_argument("--advertise-host")
help="Weight quantization for the real PyTorch backend", start_cmd.add_argument("--tracker-mode", action="store_true")
) start_cmd.add_argument("--tracker-url", default=None)
start_cmd.add_argument( start_cmd.add_argument("--wallet")
"--host", default="0.0.0.0", help="Interface to bind to" start_cmd.add_argument("--download-dir")
)
start_cmd.add_argument(
"--advertise-host",
help="Reachable host/IP to advertise to the tracker (defaults to FQDN when binding 0.0.0.0)",
)
args = parser.parse_args() args = parser.parse_args()
if args.command == "start": if args.command == "models":
from meshnet_node.startup import run_startup sys.exit(_cmd_models(args))
elif args.command == "config":
try: sys.exit(_cmd_config(args))
node = run_startup( elif args.command == "start":
tracker_url=args.tracker, sys.exit(_cmd_start(args))
port=args.port,
model=args.model,
model_id=args.model_id,
shard_start=args.shard_start,
shard_end=args.shard_end,
quantization=args.quantization,
host=args.host,
advertise_host=args.advertise_host,
)
except Exception as exc:
print(f"ERROR: {exc}", file=sys.stderr, flush=True)
sys.exit(1)
try:
while True:
time.sleep(1)
except KeyboardInterrupt:
node.stop()
sys.exit(0)
else: else:
parser.print_help() # Default: wizard or start with saved config
sys.exit(_cmd_default(args))
if __name__ == "__main__": if __name__ == "__main__":

View File

@@ -0,0 +1,72 @@
"""Persistent node configuration — stored in ~/.config/meshnet/config.json."""
from __future__ import annotations
import json
import os
import stat
from pathlib import Path
_DEFAULT_CONFIG_DIR = Path.home() / ".config" / "meshnet"
_DEFAULT_CONFIG_FILE = _DEFAULT_CONFIG_DIR / "config.json"
_DEFAULT_DOWNLOAD_DIR = Path.home() / ".meshnet" / "models"
_DEFAULT_TRACKER_URL = "http://localhost:8080"
_DEFAULT_WALLET_PATH = str(Path.home() / ".config" / "meshnet" / "wallet.json")
_DEFAULT_QUANTIZATION = "nf4"
DEFAULTS = {
"model_hf_repo": "",
"model_name": "",
"quantization": _DEFAULT_QUANTIZATION,
"download_dir": str(_DEFAULT_DOWNLOAD_DIR),
"tracker_url": _DEFAULT_TRACKER_URL,
"wallet_path": _DEFAULT_WALLET_PATH,
"shard_start": None,
"shard_end": None,
"port": 7000,
"host": "0.0.0.0",
}
def config_path(override: Path | None = None) -> Path:
return override or _DEFAULT_CONFIG_FILE
def load_config(path: Path | None = None) -> dict | None:
"""Return parsed config dict, or None if no config file exists."""
p = config_path(path)
if not p.exists():
return None
try:
cfg = json.loads(p.read_text())
if not isinstance(cfg, dict):
return None
return cfg
except (json.JSONDecodeError, OSError):
return None
def save_config(cfg: dict, path: Path | None = None) -> None:
"""Write config to disk with restricted permissions (0o600)."""
p = config_path(path)
p.parent.mkdir(parents=True, exist_ok=True)
p.write_text(json.dumps(cfg, indent=2))
try:
os.chmod(p, stat.S_IRUSR | stat.S_IWUSR)
except OSError:
pass # Windows / some filesystems don't support chmod
def delete_config(path: Path | None = None) -> None:
p = config_path(path)
if p.exists():
p.unlink()
def merge_cli_overrides(cfg: dict, **cli_kwargs) -> dict:
"""Return a copy of cfg with any non-None CLI values applied on top."""
result = dict(cfg)
for key, val in cli_kwargs.items():
if val is not None:
result[key] = val
return result

View File

@@ -0,0 +1,220 @@
"""Live node status dashboard — rich TUI with plain-text fallback."""
from __future__ import annotations
import os
import sys
import time
from collections import deque
from typing import TYPE_CHECKING
if TYPE_CHECKING:
pass
def is_interactive_tty() -> bool:
"""Return True when stdout is a real terminal (not CI / redirected / WSL2 dumb)."""
if not sys.stdout.isatty():
return False
term = os.environ.get("TERM", "")
if term in ("dumb", ""):
return False
return True
def _format_uptime(seconds: float) -> str:
s = int(seconds)
h, rem = divmod(s, 3600)
m, sec = divmod(rem, 60)
return f"{h:02d}:{m:02d}:{sec:02d}"
def _gpu_stats() -> list[dict]:
"""Return per-GPU utilization and VRAM stats, or empty list on CPU."""
try:
import torch # type: ignore[import]
if not torch.cuda.is_available():
return []
stats = []
for i in range(torch.cuda.device_count()):
props = torch.cuda.get_device_properties(i)
used = torch.cuda.memory_allocated(i)
total = props.total_memory
# Utilization requires pynvml; skip gracefully if not available
util = _nvml_gpu_util(i)
stats.append(
{
"index": i,
"name": props.name,
"used_gb": used / 1e9,
"total_gb": total / 1e9,
"util_pct": util,
}
)
return stats
except ImportError:
return []
def _nvml_gpu_util(index: int) -> int | None:
"""Return GPU utilization % via pynvml, or None if unavailable."""
try:
import pynvml # type: ignore[import]
pynvml.nvmlInit()
handle = pynvml.nvmlDeviceGetHandleByIndex(index)
rates = pynvml.nvmlDeviceGetUtilizationRates(handle)
return rates.gpu
except Exception:
return None
class _EMA:
"""Exponential moving average for tokens/sec."""
def __init__(self, alpha: float = 0.1):
self._alpha = alpha
self._value: float | None = None
def update(self, sample: float) -> float:
if self._value is None:
self._value = sample
else:
self._value = self._alpha * sample + (1 - self._alpha) * self._value
return self._value
@property
def value(self) -> float:
return self._value or 0.0
def _make_bar(pct: float, width: int = 10) -> str:
filled = round(pct / 100 * width)
return "" * filled + "" * (width - filled)
def run_dashboard(node, config: dict, start_time: float) -> None:
"""Start the live dashboard. Blocks until Ctrl-C. Returns cleanly."""
if not is_interactive_tty():
_run_plain_loop(node, config, start_time)
return
try:
from rich.live import Live # type: ignore[import]
_run_rich_dashboard(node, config, start_time)
except ImportError:
_run_plain_loop(node, config, start_time)
def _build_rich_renderable(
node, config: dict, start_time: float, tps_ema: _EMA, prev_req: list[int]
):
from rich.table import Table # type: ignore[import]
from rich.panel import Panel # type: ignore[import]
from rich.columns import Columns # type: ignore[import]
from rich.text import Text # type: ignore[import]
uptime = time.monotonic() - start_time
req_count = getattr(node, "chat_completion_count", 0)
# Tokens/sec EMA (approximate: 20 tokens per request heuristic when no real counter)
delta_req = req_count - prev_req[0]
prev_req[0] = req_count
if delta_req > 0:
approx_tokens = delta_req * 20
tps_ema.update(approx_tokens / 2.0) # 2s interval
gpu_stats = _gpu_stats()
model_name = config.get("model_name") or config.get("model_hf_repo", "unknown").split("/")[-1]
shard = ""
if config.get("shard_start") is not None:
shard = f" shard {config['shard_start']}{config['shard_end']}"
# Header line
header = Text(
f"meshnet-node {model_name} [{config.get('quantization', 'bf16')}]{shard}"
f" up {_format_uptime(uptime)}",
style="bold white",
)
# GPU table
gpu_table = Table(show_header=False, box=None, padding=(0, 1))
gpu_table.add_column("label", style="dim", no_wrap=True)
gpu_table.add_column("bar", no_wrap=True)
gpu_table.add_column("vram", no_wrap=True, style="cyan")
if gpu_stats:
for g in gpu_stats:
util = g["util_pct"]
util_str = f"{_make_bar(util)} {util:3d}%" if util is not None else " n/a"
vram_str = f"VRAM {g['used_gb']:.1f}/{g['total_gb']:.1f} GB"
gpu_table.add_row(f"GPU {g['index']} {g['name'][:20]}", util_str, vram_str)
else:
gpu_table.add_row("CPU mode", "", "no GPU detected")
# Stats panel
tps = tps_ema.value
bar_len = min(8, max(0, int(tps / 10)))
tps_bar = "▁▂▃▄▅▆▇█"[:bar_len].ljust(8)
stats_lines = [
f"Tokens/sec {tps_bar} {tps:.1f} t/s (EMA)",
f"Requests {req_count:,} served",
f"Peers 0 connected (gossip: US-017)",
f"TAI earned 0.00 TAI (payments: US-006)",
f"Uptime {_format_uptime(uptime)}",
"",
"[q] quit [c] compact view",
]
from rich.console import Group # type: ignore[import]
return Panel(
Group(header, gpu_table, Text("\n".join(stats_lines))),
title="[bold green]meshnet-node[/bold green]",
border_style="green",
)
def _run_rich_dashboard(node, config: dict, start_time: float) -> None:
from rich.live import Live # type: ignore[import]
tps_ema = _EMA()
prev_req = [0]
try:
with Live(
_build_rich_renderable(node, config, start_time, tps_ema, prev_req),
refresh_per_second=0.5,
screen=False,
) as live:
while True:
time.sleep(2)
live.update(
_build_rich_renderable(node, config, start_time, tps_ema, prev_req)
)
except KeyboardInterrupt:
pass
def _run_plain_loop(node, config: dict, start_time: float) -> None:
model_name = config.get("model_name") or config.get("model_hf_repo", "unknown").split("/")[-1]
try:
while True:
uptime = time.monotonic() - start_time
req = getattr(node, "chat_completion_count", 0)
gpu_stats = _gpu_stats()
vram_str = ""
if gpu_stats:
g = gpu_stats[0]
vram_str = f" VRAM{g['used_gb']:.1f}GB"
print(
f"[{model_name} req{req}{vram_str} up{_format_uptime(uptime)}]",
flush=True,
)
time.sleep(2)
except KeyboardInterrupt:
pass

View File

@@ -0,0 +1,165 @@
"""Curated list of models supported by the network with VRAM requirements."""
from __future__ import annotations
from dataclasses import dataclass
@dataclass
class ModelPreset:
name: str
hf_repo: str
num_layers: int
# VRAM in GB at each quantization level (None = too large to quantize this way)
vram_nf4: float
vram_int8: float
vram_bf16: float
description: str
def vram_for_quant(self, quant: str) -> float:
"""Return VRAM requirement in GB for the given quantization."""
q = quant.lower().replace("bfloat16", "bf16")
if q == "nf4":
return self.vram_nf4
if q in ("int8", "int8"):
return self.vram_int8
if q in ("bf16", "bfloat16"):
return self.vram_bf16
raise ValueError(f"unknown quantization: {quant!r}")
def fits_vram(self, available_gb: float, quant: str) -> bool:
return self.vram_for_quant(quant) <= available_gb
def recommended_quant(self, available_gb: float) -> str | None:
"""Return the highest-quality quantization that fits available VRAM, or None."""
if self.vram_bf16 <= available_gb:
return "bf16"
if self.vram_int8 <= available_gb:
return "int8"
if self.vram_nf4 <= available_gb:
return "nf4"
return None
CURATED_MODELS: list[ModelPreset] = [
ModelPreset(
name="Qwen2.5-0.5B-Instruct",
hf_repo="Qwen/Qwen2.5-0.5B-Instruct",
num_layers=24,
vram_nf4=0.4,
vram_int8=0.6,
vram_bf16=1.0,
description="Smallest no-gating model — great for testing, ~1 GB",
),
ModelPreset(
name="Qwen2.5-1.5B-Instruct",
hf_repo="Qwen/Qwen2.5-1.5B-Instruct",
num_layers=28,
vram_nf4=1.0,
vram_int8=1.8,
vram_bf16=3.2,
description="Fast no-gating model — good quality, ~3 GB",
),
ModelPreset(
name="Llama-3-70B-Instruct",
hf_repo="meta-llama/Meta-Llama-3-70B-Instruct",
num_layers=80,
vram_nf4=18.0,
vram_int8=40.0,
vram_bf16=140.0,
description="Meta's flagship 70B instruction model",
),
ModelPreset(
name="Qwen2.5-72B-Instruct",
hf_repo="Qwen/Qwen2.5-72B-Instruct",
num_layers=80,
vram_nf4=19.0,
vram_int8=41.0,
vram_bf16=145.0,
description="Alibaba's 72B multilingual instruction model",
),
ModelPreset(
name="Mixtral-8x7B-Instruct",
hf_repo="mistralai/Mixtral-8x7B-Instruct-v0.1",
num_layers=32,
vram_nf4=7.0,
vram_int8=14.0,
vram_bf16=27.0,
description="Mistral's sparse MoE — fast and efficient",
),
ModelPreset(
name="Llama-3-8B-Instruct",
hf_repo="meta-llama/Meta-Llama-3-8B-Instruct",
num_layers=32, # gated repo — requires HF login
vram_nf4=4.5,
vram_int8=8.5,
vram_bf16=16.0,
description="Meta's compact 8B model — good for low-VRAM nodes",
),
ModelPreset(
name="Phi-3-medium-128k",
hf_repo="microsoft/Phi-3-medium-128k-instruct",
num_layers=40,
vram_nf4=4.0,
vram_int8=8.0,
vram_bf16=15.0,
description="Microsoft's efficient 14B model with 128k context",
),
ModelPreset(
name="Gemma-2-27B-IT",
hf_repo="google/gemma-2-27b-it",
num_layers=46,
vram_nf4=10.0,
vram_int8=20.0,
vram_bf16=54.0,
description="Google's 27B instruction-tuned model",
),
ModelPreset(
name="DeepSeek-V2-Lite-Chat",
hf_repo="deepseek-ai/DeepSeek-V2-Lite-Chat",
num_layers=27,
vram_nf4=5.0,
vram_int8=9.0,
vram_bf16=16.0,
description="DeepSeek's efficient MoE — strong coding + reasoning",
),
]
def detect_num_layers(hf_repo: str) -> int | None:
"""Return num_hidden_layers from HuggingFace config.json (downloads ~1 KB only)."""
# Check curated list first (no network call)
for m in CURATED_MODELS:
if m.hf_repo == hf_repo:
return m.num_layers
try:
from transformers import AutoConfig # type: ignore[import]
cfg = AutoConfig.from_pretrained(hf_repo)
return int(cfg.num_hidden_layers)
except Exception:
return None
def browse_hf_hub(top_n: int = 20) -> list[dict]:
"""Fetch top downloaded text-generation models from HuggingFace Hub."""
try:
from huggingface_hub import list_models # type: ignore[import]
models = list(
list_models(
pipeline_tag="text-generation",
library="transformers",
sort="downloads",
direction=-1,
limit=top_n,
)
)
return [
{
"repo": m.id,
"downloads": getattr(m, "downloads", 0) or 0,
}
for m in models
]
except Exception as exc:
raise RuntimeError(f"HuggingFace Hub lookup failed: {exc}") from exc

View File

@@ -2,6 +2,7 @@
import http.server import http.server
import json import json
import time
import threading import threading
import urllib.parse import urllib.parse
from pathlib import Path from pathlib import Path
@@ -93,6 +94,8 @@ class _StubHTTPServer(http.server.HTTPServer):
response_prefix: str, response_prefix: str,
model: str, model: str,
shard_path: Path | None, shard_path: Path | None,
tracker_mode: bool = False,
tracker_url: str | None = None,
): ):
super().__init__(addr, handler) super().__init__(addr, handler)
self.shard_start = shard_start self.shard_start = shard_start
@@ -103,6 +106,9 @@ class _StubHTTPServer(http.server.HTTPServer):
self.shard_path = shard_path self.shard_path = shard_path
self.received_activations: bool = False self.received_activations: bool = False
self.forward_chunk_count: int = 0 self.forward_chunk_count: int = 0
self.tracker_mode: bool = tracker_mode
self.tracker_url: str | None = tracker_url
self.chat_completion_count: int = 0
class _StubHandler(http.server.BaseHTTPRequestHandler): class _StubHandler(http.server.BaseHTTPRequestHandler):
@@ -110,10 +116,13 @@ class _StubHandler(http.server.BaseHTTPRequestHandler):
pass pass
def do_POST(self): def do_POST(self):
server: _StubHTTPServer = self.server # type: ignore[assignment]
if self.path == "/v1/infer": if self.path == "/v1/infer":
self._handle_infer() self._handle_infer()
elif self.path == "/forward": elif self.path == "/forward":
self._handle_forward() self._handle_forward()
elif self.path == "/v1/chat/completions" and server.tracker_mode:
self._handle_chat_completions()
else: else:
self.send_response(404) self.send_response(404)
self.end_headers() self.end_headers()
@@ -126,6 +135,82 @@ class _StubHandler(http.server.BaseHTTPRequestHandler):
self.send_response(404) self.send_response(404)
self.end_headers() self.end_headers()
def _send_json(self, status: int, data: dict) -> None:
payload = json.dumps(data).encode()
self.send_response(status)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(payload)))
self.end_headers()
self.wfile.write(payload)
def _handle_chat_completions(self) -> None:
server: _StubHTTPServer = self.server # type: ignore[assignment]
length = int(self.headers.get("Content-Length", 0))
try:
body = json.loads(self.rfile.read(length) or b"{}")
except (json.JSONDecodeError, ValueError):
self._send_json(400, {"error": "invalid JSON body"})
return
if not isinstance(body, dict):
self._send_json(400, {"error": "JSON body must be an object"})
return
server.chat_completion_count += 1
streaming = bool(body.get("stream", False))
model = str(body.get("model", server.model))
messages = body.get("messages", [])
last_content = ""
if isinstance(messages, list) and messages:
last = messages[-1]
if isinstance(last, dict):
last_content = str(last.get("content", ""))
text = f"{server.response_prefix} {last_content}"
if streaming:
self._send_sse_response(text, model)
else:
created = int(time.time())
self._send_json(200, {
"id": "chatcmpl-stub",
"object": "chat.completion",
"created": created,
"model": model,
"choices": [{
"index": 0,
"message": {"role": "assistant", "content": text},
"finish_reason": "stop",
}],
"usage": {"prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0},
})
def _send_sse_response(self, text: str, model: str) -> None:
chunk_id = "chatcmpl-stub"
created = int(time.time())
self.send_response(200)
self.send_header("Content-Type", "text/event-stream; charset=utf-8")
self.send_header("Cache-Control", "no-cache")
self.end_headers()
def _emit(data: str) -> None:
self.wfile.write(f"data: {data}\n\n".encode())
self.wfile.flush()
_emit(json.dumps({
"id": chunk_id, "object": "chat.completion.chunk", "created": created,
"model": model,
"choices": [{"index": 0, "delta": {"role": "assistant", "content": ""}, "finish_reason": None}],
}))
_emit(json.dumps({
"id": chunk_id, "object": "chat.completion.chunk", "created": created,
"model": model,
"choices": [{"index": 0, "delta": {"content": text}, "finish_reason": None}],
}))
_emit(json.dumps({
"id": chunk_id, "object": "chat.completion.chunk", "created": created,
"model": model,
"choices": [{"index": 0, "delta": {}, "finish_reason": "stop"}],
}))
self.wfile.write(b"data: [DONE]\n\n")
self.wfile.flush()
def _handle_shard_download(self, parsed: urllib.parse.ParseResult): def _handle_shard_download(self, parsed: urllib.parse.ParseResult):
server: _StubHTTPServer = self.server # type: ignore[assignment] server: _StubHTTPServer = self.server # type: ignore[assignment]
params = urllib.parse.parse_qs(parsed.query) params = urllib.parse.parse_qs(parsed.query)
@@ -246,6 +331,7 @@ class StubNodeServer:
shard_start / shard_end define which transformer layer range this node owns. shard_start / shard_end define which transformer layer range this node owns.
is_last_shard controls whether the node returns a text response (True) or is_last_shard controls whether the node returns a text response (True) or
activation tensors (False) after processing its shard. activation tensors (False) after processing its shard.
tracker_mode enables the /v1/chat/completions endpoint for head-shard nodes.
""" """
def __init__( def __init__(
@@ -258,6 +344,8 @@ class StubNodeServer:
response_prefix: str = "stub response to:", response_prefix: str = "stub response to:",
model: str = "stub-model", model: str = "stub-model",
shard_path: Path | None = None, shard_path: Path | None = None,
tracker_mode: bool = False,
tracker_url: str | None = None,
): ):
self._host = host self._host = host
self._requested_port = port self._requested_port = port
@@ -269,6 +357,8 @@ class StubNodeServer:
self._response_prefix = response_prefix self._response_prefix = response_prefix
self._model = model self._model = model
self._shard_path = shard_path self._shard_path = shard_path
self._tracker_mode = tracker_mode
self._tracker_url = tracker_url
self._server: _StubHTTPServer | None = None self._server: _StubHTTPServer | None = None
self._thread: threading.Thread | None = None self._thread: threading.Thread | None = None
self.port: int | None = None self.port: int | None = None
@@ -283,6 +373,11 @@ class StubNodeServer:
"""Number of binary /forward chunks handled since this node was started.""" """Number of binary /forward chunks handled since this node was started."""
return self._server.forward_chunk_count if self._server is not None else 0 return self._server.forward_chunk_count if self._server is not None else 0
@property
def chat_completion_count(self) -> int:
"""Number of /v1/chat/completions requests handled since this node was started."""
return self._server.chat_completion_count if self._server is not None else 0
def start(self) -> int: def start(self) -> int:
if self._server is not None: if self._server is not None:
raise RuntimeError("StubNodeServer is already running") raise RuntimeError("StubNodeServer is already running")
@@ -296,6 +391,8 @@ class StubNodeServer:
self._response_prefix, self._response_prefix,
self._model, self._model,
self._shard_path, self._shard_path,
self._tracker_mode,
self._tracker_url,
) )
self.port = self._server.server_address[1] self.port = self._server.server_address[1]
self._thread = threading.Thread(target=self._server.serve_forever, daemon=True) self._thread = threading.Thread(target=self._server.serve_forever, daemon=True)

View File

@@ -84,7 +84,19 @@ def run_startup(
if probationary_line is not None: if probationary_line is not None:
print(f" {probationary_line}", flush=True) print(f" {probationary_line}", flush=True)
if model_id is not None and shard_start is not None and shard_end is not None: if model_id is not None:
# Auto-detect shard range from model config if not explicitly provided
if shard_start is None or shard_end is None:
detected = _detect_num_layers(model_id)
if detected is None:
raise ValueError(
f"Could not read num_hidden_layers from {model_id} config. "
"Pass --shard-start and --shard-end explicitly."
)
shard_start = shard_start if shard_start is not None else 0
shard_end = shard_end if shard_end is not None else detected - 1
print(f" Auto-detected {detected} layers → shard {shard_start}{shard_end}", flush=True)
print("Loading real PyTorch model shard...", flush=True) print("Loading real PyTorch model shard...", flush=True)
node = TorchNodeServer( node = TorchNodeServer(
host=host, host=host,
@@ -102,7 +114,7 @@ def run_startup(
f"meshnet-node ready\n" f"meshnet-node ready\n"
f" Wallet: {address}\n" f" Wallet: {address}\n"
f" Model ID: {model_id}\n" f" Model ID: {model_id}\n"
f" Shard: layers {shard_start}-{shard_end}\n" f" Shard: layers {shard_start}{shard_end}\n"
f" Quantization: {quantization}\n" f" Quantization: {quantization}\n"
f" Endpoint: {endpoint}\n" f" Endpoint: {endpoint}\n"
f" Hardware: {device.upper()}\n" f" Hardware: {device.upper()}\n"
@@ -110,8 +122,8 @@ def run_startup(
flush=True, flush=True,
) )
return node return node
if model_id is not None or shard_start is not None or shard_end is not None: if shard_start is not None or shard_end is not None:
raise ValueError("--model-id, --shard-start, and --shard-end must be provided together") raise ValueError("--shard-start / --shard-end require --model-id")
# 3. Shard assignment from tracker # 3. Shard assignment from tracker
print("Querying tracker for shard assignment...", flush=True) print("Querying tracker for shard assignment...", flush=True)
@@ -201,6 +213,17 @@ def run_startup(
return node return node
def _detect_num_layers(model_id: str) -> int | None:
"""Fetch num_hidden_layers from HuggingFace model config (downloads ~1 KB config.json only)."""
try:
from transformers import AutoConfig # type: ignore[import]
cfg = AutoConfig.from_pretrained(model_id)
return int(cfg.num_hidden_layers)
except Exception as exc:
print(f" Warning: could not read model config from HF: {exc}", flush=True)
return None
def _probationary_status_line(contracts: Any | None, wallet_address: str) -> str | None: def _probationary_status_line(contracts: Any | None, wallet_address: str) -> str | None:
if contracts is None: if contracts is None:
return None return None

View File

@@ -6,6 +6,11 @@ import http.server
import json import json
import sys import sys
import threading import threading
import time
import urllib.error
import urllib.parse
import urllib.request
import uuid
from .model_backend import ( from .model_backend import (
InsufficientVRAMError, InsufficientVRAMError,
@@ -24,11 +29,20 @@ from .server import (
class _TorchHTTPServer(http.server.HTTPServer): class _TorchHTTPServer(http.server.HTTPServer):
def __init__(self, addr, handler, backend: TorchModelShard): def __init__(
self,
addr,
handler,
backend: TorchModelShard,
tracker_mode: bool = False,
tracker_url: str | None = None,
):
super().__init__(addr, handler) super().__init__(addr, handler)
self.backend = backend self.backend = backend
self.received_activations = False self.received_activations = False
self.forward_chunk_count = 0 self.forward_chunk_count = 0
self.tracker_mode = tracker_mode
self.tracker_url = tracker_url
class _TorchHandler(http.server.BaseHTTPRequestHandler): class _TorchHandler(http.server.BaseHTTPRequestHandler):
@@ -36,10 +50,13 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
pass pass
def do_POST(self): def do_POST(self):
server: _TorchHTTPServer = self.server # type: ignore[assignment]
if self.path == "/forward": if self.path == "/forward":
self._handle_forward() self._handle_forward()
elif self.path == "/v1/infer": elif self.path == "/v1/infer":
self._handle_infer() self._handle_infer()
elif self.path == "/v1/chat/completions" and server.tracker_mode:
self._handle_chat_completions()
else: else:
self.send_response(404) self.send_response(404)
self.end_headers() self.end_headers()
@@ -190,6 +207,152 @@ class _TorchHandler(http.server.BaseHTTPRequestHandler):
self.end_headers() self.end_headers()
self.wfile.write(payload) self.wfile.write(payload)
def _handle_chat_completions(self) -> None:
server: _TorchHTTPServer = self.server # type: ignore[assignment]
body = self._read_json_body()
if body is None:
return
messages = body.get("messages", [])
stream = bool(body.get("stream", False))
model = str(body.get("model", ""))
prompt = " ".join(
str(m.get("content", ""))
for m in messages
if isinstance(m, dict) and m.get("role") == "user"
)
try:
payload = server.backend.encode_prompt(prompt)
except Exception as exc:
self._send_json(500, {"error": f"encode_prompt failed: {exc}"})
return
remaining_route = self._get_remaining_route(model)
result_text = self._run_downstream_pipeline(payload, remaining_route)
self._send_openai_response(result_text, model, stream)
def _get_remaining_route(self, model: str) -> list[str]:
server: _TorchHTTPServer = self.server # type: ignore[assignment]
if server.tracker_url is None:
return []
try:
url = f"{server.tracker_url}/v1/route?model={urllib.parse.quote(model)}"
with urllib.request.urlopen(url, timeout=5.0) as r:
route_resp = json.loads(r.read())
route = route_resp.get("route", [])
# Skip the first node in the route (self) since we're already the head
return list(route[1:])
except Exception:
return []
def _run_downstream_pipeline(self, payload: object, route: list[str]) -> str:
server: _TorchHTTPServer = self.server # type: ignore[assignment]
if not route:
# Single-node mode: decode tail locally if we're the tail
if server.backend.is_tail:
try:
tensor = server.backend.torch.frombuffer(
bytearray(payload.body), # type: ignore[union-attr]
dtype=server.backend.torch.bfloat16,
).reshape(payload.shape).to(server.backend.device) # type: ignore[union-attr]
return server.backend.decode_tail(tensor)
except Exception as exc:
return f"decode error: {exc}"
return ""
session = str(uuid.uuid4())
shape = payload.shape # type: ignore[union-attr]
attn_mask = payload.attention_mask_header # type: ignore[union-attr]
pos_ids = payload.position_ids_header # type: ignore[union-attr]
current_body = payload.body # type: ignore[union-attr]
current_shape = shape
current_attn = attn_mask
current_pos = pos_ids
for hop_index, node_url in enumerate(route):
headers: dict[str, str] = {
"Content-Type": "application/octet-stream",
"X-Meshnet-Wire": _WIRE_VERSION,
"X-Meshnet-Shape": ",".join(str(d) for d in current_shape),
"X-Meshnet-Dtype": "bfloat16",
"X-Meshnet-Session": session,
"X-Meshnet-Chunk-Index": "0",
"X-Meshnet-Chunk-Total": "1",
"X-Meshnet-Hop-Index": str(hop_index),
}
if current_attn:
headers["X-Meshnet-Attn-Mask"] = current_attn
if current_pos:
headers["X-Meshnet-Position-Ids"] = current_pos
req = urllib.request.Request(
f"{node_url}/forward",
data=current_body,
headers=headers,
method="POST",
)
try:
with urllib.request.urlopen(req, timeout=10.0) as r:
resp_body = r.read()
resp_headers = {k.lower(): v for k, v in r.headers.items()}
except Exception as exc:
return f"pipeline error at {node_url}: {exc}"
content_type = resp_headers.get("content-type", "")
if "application/json" in content_type:
try:
data = json.loads(resp_body)
return str(data.get("text", ""))
except json.JSONDecodeError:
return resp_body.decode("utf-8", errors="replace")
# Binary activation — update and forward to next node
shape_header = resp_headers.get("x-meshnet-shape", ",".join(str(d) for d in current_shape))
current_shape = _parse_shape(shape_header)
current_body = resp_body
current_attn = resp_headers.get("x-meshnet-attn-mask")
current_pos = resp_headers.get("x-meshnet-position-ids")
return ""
def _send_openai_response(self, text: str, model: str, stream: bool) -> None:
chunk_id = "chatcmpl-node"
created = int(time.time())
if not stream:
self._send_json(200, {
"id": chunk_id,
"object": "chat.completion",
"created": created,
"model": model,
"choices": [{
"index": 0,
"message": {"role": "assistant", "content": text},
"finish_reason": "stop",
}],
"usage": {"prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0},
})
return
self.send_response(200)
self.send_header("Content-Type", "text/event-stream; charset=utf-8")
self.send_header("Cache-Control", "no-cache")
self.end_headers()
def _emit(data: str) -> None:
self.wfile.write(f"data: {data}\n\n".encode())
self.wfile.flush()
_emit(json.dumps({
"id": chunk_id, "object": "chat.completion.chunk", "created": created,
"model": model,
"choices": [{"index": 0, "delta": {"role": "assistant", "content": ""}, "finish_reason": None}],
}))
_emit(json.dumps({
"id": chunk_id, "object": "chat.completion.chunk", "created": created,
"model": model,
"choices": [{"index": 0, "delta": {"content": text}, "finish_reason": None}],
}))
_emit(json.dumps({
"id": chunk_id, "object": "chat.completion.chunk", "created": created,
"model": model,
"choices": [{"index": 0, "delta": {}, "finish_reason": "stop"}],
}))
self.wfile.write(b"data: [DONE]\n\n")
self.wfile.flush()
class TorchNodeServer: class TorchNodeServer:
"""HTTP server backed by a HuggingFace causal language model shard.""" """HTTP server backed by a HuggingFace causal language model shard."""
@@ -203,6 +366,8 @@ class TorchNodeServer:
shard_end: int = 6, shard_end: int = 6,
quantization: str = "bfloat16", quantization: str = "bfloat16",
backend: TorchModelShard | None = None, backend: TorchModelShard | None = None,
tracker_mode: bool | None = None,
tracker_url: str | None = None,
) -> None: ) -> None:
self._host = host self._host = host
self._requested_port = port self._requested_port = port
@@ -212,6 +377,9 @@ class TorchNodeServer:
shard_end, shard_end,
quantization, quantization,
) )
# Auto-detect tracker mode: enabled when shard_start == 0 or explicitly set
self._tracker_mode = tracker_mode if tracker_mode is not None else (shard_start == 0)
self._tracker_url = tracker_url
self._server: _TorchHTTPServer | None = None self._server: _TorchHTTPServer | None = None
self._thread: threading.Thread | None = None self._thread: threading.Thread | None = None
self.port: int | None = None self.port: int | None = None
@@ -235,6 +403,8 @@ class TorchNodeServer:
(self._host, self._requested_port), (self._host, self._requested_port),
_TorchHandler, _TorchHandler,
self._backend, self._backend,
self._tracker_mode,
self._tracker_url,
) )
self.port = self._server.server_address[1] self.port = self._server.server_address[1]
self._thread = threading.Thread(target=self._server.serve_forever, daemon=True) self._thread = threading.Thread(target=self._server.serve_forever, daemon=True)

View File

@@ -0,0 +1,332 @@
"""Interactive first-run setup wizard — mining-client style."""
from __future__ import annotations
import sys
import urllib.error
import urllib.request
from pathlib import Path
from typing import TYPE_CHECKING
from .config import DEFAULTS, _DEFAULT_DOWNLOAD_DIR, _DEFAULT_TRACKER_URL, _DEFAULT_WALLET_PATH
from .model_catalog import CURATED_MODELS, ModelPreset, browse_hf_hub, detect_num_layers
if TYPE_CHECKING:
pass
_HEADER = """\
╔══════════════════════════════════════════════════════════════════╗
║ meshnet-node v0.1.0 ║
║ Distributed AI Inference — Node Setup ║
╚══════════════════════════════════════════════════════════════════╝
"""
_QUANT_LABELS = {"nf4": "NF4 (4-bit)", "int8": "INT8 (8-bit)", "bf16": "BF16 (full)"}
def _ask(prompt: str, default: str = "", validator=None) -> str:
"""Prompt user and return answer. Returns default on empty input or EOF."""
display = f"{prompt} [{default}]: " if default else f"{prompt}: "
while True:
try:
raw = input(display).strip()
except (EOFError, KeyboardInterrupt):
print()
raise KeyboardInterrupt
value = raw or default
if validator is None or validator(value):
return value
# validator returned error string
print(f"{validator(value)}")
def _ask_int(prompt: str, default: int, lo: int, hi: int) -> int:
def validate(s: str) -> bool | str:
try:
v = int(s)
except ValueError:
return "Please enter a number."
if not (lo <= v <= hi):
return f"Please enter a number between {lo} and {hi}."
return True
while True:
raw = _ask(prompt, str(default))
try:
v = int(raw)
if lo <= v <= hi:
return v
except ValueError:
pass
print(f" ✗ Enter a number between {lo} and {hi}.")
def _ask_yn(prompt: str, default: bool = True) -> bool:
hint = "Y/n" if default else "y/N"
raw = _ask(f"{prompt} [{hint}]").lower()
if not raw:
return default
return raw.startswith("y")
def _detect_gpus() -> list[dict]:
"""Return list of detected GPU dicts with name and vram_gb."""
gpus: list[dict] = []
try:
import torch # type: ignore[import]
if torch.cuda.is_available():
for i in range(torch.cuda.device_count()):
props = torch.cuda.get_device_properties(i)
gpus.append(
{
"index": i,
"name": props.name,
"vram_gb": props.total_memory / 1e9,
"backend": "cuda",
}
)
except ImportError:
pass
return gpus
def _total_vram_gb(gpus: list[dict]) -> float:
return sum(g["vram_gb"] for g in gpus)
def _print_gpus(gpus: list[dict]) -> None:
if not gpus:
print(" ⚠ No CUDA GPU detected — running in CPU mode")
print(" CPU inference is very slow. Consider a machine with an NVIDIA GPU.")
return
for g in gpus:
vram = g["vram_gb"]
print(f" GPU {g['index']}: {g['name']} {vram:.0f} GB VRAM ✓")
def _print_model_table(gpus: list[dict], quant: str = "nf4") -> None:
available_gb = _total_vram_gb(gpus)
print()
print(f" # {'Model':<30} {'Layers':>6} {'NF4':>6} {'INT8':>6} {'BF16':>6}")
print(f" {''*4} {''*30} {''*6} {''*6} {''*6} {''*6}")
for i, m in enumerate(CURATED_MODELS, 1):
fits_nf4 = "" if m.vram_nf4 <= available_gb else ""
fits_int8 = "" if m.vram_int8 <= available_gb else ""
fits_bf16 = "" if m.vram_bf16 <= available_gb else ""
nf4_str = f"{fits_nf4}{m.vram_nf4:.0f}GB"
int8_str = f"{fits_int8}{m.vram_int8:.0f}GB"
bf16_str = f"{fits_bf16}{m.vram_bf16:.0f}GB"
print(f" {i:<3} {m.name:<30} {m.num_layers:>6} {nf4_str:>6} {int8_str:>6} {bf16_str:>6}")
print(f" {m.description}")
idx = len(CURATED_MODELS) + 1
print(f" {idx:<3} {'[Browse HuggingFace Hub...]':<30}")
print()
def _browse_hf_interactive() -> str | None:
"""Show HF Hub top-20 and let user enter a repo ID. Returns repo ID or None to go back."""
print("\nFetching top models from HuggingFace Hub...")
try:
models = browse_hf_hub(top_n=20)
except RuntimeError as exc:
print(f"{exc}")
return None
print(f"\n {'#':<4} {'HuggingFace Repo':<50} Downloads")
print(f" {''*4} {''*50} {''*10}")
for i, m in enumerate(models, 1):
dl = m["downloads"]
dl_str = f"{dl/1e6:.1f}M" if dl >= 1_000_000 else f"{dl/1e3:.0f}k" if dl >= 1000 else str(dl)
print(f" {i:<4} {m['repo']:<50} {dl_str}")
print()
raw = _ask(
"Enter a number to select, or paste any HuggingFace repo ID (or press Enter to go back)",
default="",
)
if not raw:
return None
try:
idx = int(raw) - 1
if 0 <= idx < len(models):
return models[idx]["repo"]
except ValueError:
pass
# Treat raw input as a repo ID
if "/" in raw:
return raw
print(" ✗ Invalid input — please enter a number or a full repo ID like 'org/model-name'")
return None
def _ask_quant(gpus: list[dict], model: ModelPreset | None) -> str:
available_gb = _total_vram_gb(gpus)
print("\nQuantization level:")
options: list[tuple[str, str]] = []
for quant, label in [("nf4", "NF4 4-bit"), ("int8", "INT8 8-bit"), ("bf16", "BF16 full precision")]:
if model is not None:
vram = model.vram_for_quant(quant)
fits = "" if vram <= available_gb else "✗ insufficient VRAM"
suffix = f" ({vram:.0f} GB needed — {fits})"
else:
suffix = ""
options.append((quant, f"{label}{suffix}"))
for i, (_, label) in enumerate(options, 1):
print(f" {i}) {label}")
# Recommend the best fitting quant
if model is not None:
rec = model.recommended_quant(available_gb)
rec_idx = next((i for i, (q, _) in enumerate(options, 1) if q == rec), 1) if rec else 1
default_idx = rec_idx
print(f" (Recommended: {rec.upper() if rec else 'NF4'} for your GPU)")
else:
default_idx = 1
choice = _ask_int("Enter number", default_idx, 1, 3)
return options[choice - 1][0]
def _validate_dir(path_str: str) -> bool | str:
p = Path(path_str).expanduser()
try:
p.mkdir(parents=True, exist_ok=True)
return True
except OSError as exc:
return f"Cannot create directory: {exc}"
def _validate_tracker(url: str) -> bool | str:
if not url.startswith(("http://", "https://")):
return "URL must start with http:// or https://"
return True
def _ping_tracker(url: str) -> bool:
"""Return True if tracker responds to /health."""
try:
with urllib.request.urlopen(f"{url.rstrip('/')}/health", timeout=3):
return True
except Exception:
return False
def run_wizard(config_path_override=None) -> dict:
"""Run the interactive setup wizard and return a config dict.
Raises KeyboardInterrupt if user presses Ctrl-C.
"""
print(_HEADER)
# Step 1: GPU detection
print("Detecting hardware...")
gpus = _detect_gpus()
_print_gpus(gpus)
available_gb = _total_vram_gb(gpus)
if available_gb == 0:
available_gb = 9999 # CPU — don't filter models by VRAM
# Step 2 & 3: Model selection
print("\nSelect a model to serve:\n")
selected_repo: str | None = None
selected_preset: ModelPreset | None = None
while selected_repo is None:
_print_model_table(gpus)
lo, hi = 1, len(CURATED_MODELS) + 1
choice = _ask_int("Enter number", 1, lo, hi)
if choice == len(CURATED_MODELS) + 1:
repo = _browse_hf_interactive()
if repo:
# Look up layer count for custom repo
print(f" Checking {repo} config...", end=" ", flush=True)
layers = detect_num_layers(repo)
if layers:
print(f"{layers} layers")
else:
print("(layer count unknown — will detect on start)")
selected_repo = repo
selected_preset = None
else:
selected_preset = CURATED_MODELS[choice - 1]
selected_repo = selected_preset.hf_repo
if selected_preset.recommended_quant(available_gb) is None:
print(
f"\n ⚠ Warning: {selected_preset.name} requires at least "
f"{selected_preset.vram_nf4:.0f} GB VRAM at NF4 — even the smallest "
f"quantization may be too large for your GPU."
)
if not _ask_yn("Continue anyway?", default=False):
selected_repo = None
selected_preset = None
num_layers = (selected_preset.num_layers if selected_preset
else detect_num_layers(selected_repo or ""))
layers_str = f" {num_layers} layers" if num_layers else ""
print(f"\n ✓ Selected: {selected_repo}{layers_str}")
# Step 3b: Quantization
quant = _ask_quant(gpus, selected_preset)
print(f" ✓ Quantization: {quant.upper()}")
# Step 4: Download directory
print()
dl_dir = _ask(
"Download directory",
default=str(_DEFAULT_DOWNLOAD_DIR),
validator=lambda v: _validate_dir(v) if v else "Directory is required.",
)
print(f" ✓ Download dir: {dl_dir}")
# Step 5: Tracker URL
print()
tracker_url = _DEFAULT_TRACKER_URL
raw_tracker = _ask("Tracker URL", default=_DEFAULT_TRACKER_URL, validator=_validate_tracker)
tracker_url = raw_tracker
if _ping_tracker(tracker_url):
print(f" ✓ Tracker reachable: {tracker_url}")
else:
print(f" ⚠ Tracker not reachable at {tracker_url} (will retry on start)")
# Step 6: Wallet path
print()
wallet_path = _ask("Wallet path", default=_DEFAULT_WALLET_PATH)
print(f" ✓ Wallet: {wallet_path}")
cfg = {
"model_hf_repo": selected_repo,
"model_name": selected_preset.name if selected_preset else selected_repo.split("/")[-1],
"quantization": quant,
"download_dir": dl_dir,
"tracker_url": tracker_url,
"wallet_path": wallet_path,
"shard_start": None,
"shard_end": None,
"port": DEFAULTS["port"],
"host": DEFAULTS["host"],
}
return cfg
def print_models_table(available_gb: float | None = None) -> None:
"""Print curated model table for `meshnet-node models`."""
gpus: list[dict] = []
if available_gb is None:
gpus = _detect_gpus()
available_gb = _total_vram_gb(gpus) or 9999
else:
gpus = [{"index": 0, "name": "GPU", "vram_gb": available_gb, "backend": "cuda"}]
print(f"\n{'#':<4} {'Model':<32} {'HuggingFace Repo':<45} {'Layers':>6} {'NF4':>8} {'INT8':>8} {'BF16':>8}")
print(f"{''*4} {''*32} {''*45} {''*6} {''*8} {''*8} {''*8}")
for i, m in enumerate(CURATED_MODELS, 1):
def _cell(vram: float) -> str:
fits = "" if vram <= available_gb else ""
return f"{fits}{vram:.0f}GB"
print(
f"{i:<4} {m.name:<32} {m.hf_repo:<45} {m.num_layers:>6} "
f"{_cell(m.vram_nf4):>8} {_cell(m.vram_int8):>8} {_cell(m.vram_bf16):>8}"
)
print()

View File

@@ -13,6 +13,7 @@ dependencies = [
"huggingface-hub>=0.20", "huggingface-hub>=0.20",
"accelerate>=0.28", "accelerate>=0.28",
"bitsandbytes>=0.43", "bitsandbytes>=0.43",
"rich>=13",
"safetensors>=0.4", "safetensors>=0.4",
"torch>=2.1", "torch>=2.1",
"transformers>=4.39", "transformers>=4.39",

View File

@@ -0,0 +1,162 @@
"""WebSocket gossip client — connects to relay, publish/subscribe to topics."""
from __future__ import annotations
import asyncio
import json
import logging
import threading
import time
import uuid
from collections import defaultdict
from typing import Callable
log = logging.getLogger(__name__)
# Message envelope topics
TOPIC_NODE_JOIN = "node-join"
TOPIC_NODE_LEAVE = "node-leave"
TOPIC_COVERAGE_UPDATE = "coverage-update"
TOPIC_HEARTBEAT = "heartbeat"
TOPIC_PEER_LIST = "peer-list"
TOPIC_RELAY_ANNOUNCE = "relay-announce"
_MSG_TTL = 3 # max re-broadcast hops
def _make_envelope(topic: str, payload: dict, from_peer: str, ttl: int = _MSG_TTL) -> dict:
return {
"topic": topic,
"version": 1,
"from_peer": from_peer,
"timestamp": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
"msg_id": str(uuid.uuid4()),
"ttl": ttl,
"payload": payload,
}
class GossipClient:
"""Thread-safe WebSocket gossip client.
Usage::
client = GossipClient(relay_url="ws://relay:8765", peer_id="abc123")
client.subscribe("node-join", lambda env: print(env["payload"]))
client.start()
client.publish("node-join", {"addr": "http://192.168.1.42:8001", ...})
...
client.stop()
"""
def __init__(self, relay_url: str, peer_id: str, reconnect_interval: float = 3.0):
self.relay_url = relay_url
self.peer_id = peer_id
self.reconnect_interval = reconnect_interval
self._handlers: dict[str, list[Callable]] = defaultdict(list)
self._seen: set[str] = set()
self._loop: asyncio.AbstractEventLoop | None = None
self._thread: threading.Thread | None = None
self._ws = None
self._running = False
self._connected = threading.Event()
self._stop_event: asyncio.Event | None = None
def subscribe(self, topic: str, handler: Callable) -> None:
"""Register a sync callback for messages on topic."""
self._handlers[topic].append(handler)
def publish(self, topic: str, payload: dict) -> None:
"""Send a gossip message to all peers via the relay. Thread-safe."""
envelope = _make_envelope(topic, payload, self.peer_id)
if self._loop and self._running:
asyncio.run_coroutine_threadsafe(self._send(envelope), self._loop)
def start(self) -> None:
"""Start the gossip client in a background thread."""
self._running = True
self._loop = asyncio.new_event_loop()
self._thread = threading.Thread(target=self._run_loop, daemon=True, name="gossip")
self._thread.start()
def wait_connected(self, timeout: float = 5.0) -> bool:
"""Block until connected to relay or timeout. Returns True if connected."""
return self._connected.wait(timeout)
def stop(self) -> None:
"""Stop the gossip client and clean up."""
self._running = False
if self._loop and self._stop_event is not None:
self._loop.call_soon_threadsafe(self._stop_event.set)
if self._thread:
self._thread.join(timeout=3.0)
# ------------------------------------------------------------------
# Internal asyncio methods (run inside the background event loop)
# ------------------------------------------------------------------
def _run_loop(self) -> None:
asyncio.set_event_loop(self._loop)
self._stop_event = asyncio.Event()
try:
self._loop.run_until_complete(self._connect_loop())
except Exception:
log.debug("Gossip loop exited", exc_info=True)
async def _connect_loop(self) -> None:
import websockets # type: ignore[import]
while self._running and not (self._stop_event and self._stop_event.is_set()):
try:
async with websockets.connect(
self.relay_url,
ping_interval=20,
ping_timeout=10,
open_timeout=5,
) as ws:
self._ws = ws
self._connected.set()
log.debug("Gossip connected to %s", self.relay_url)
# Send peer registration
await ws.send(json.dumps(
_make_envelope(
"peer-register",
{"peer_id": self.peer_id},
self.peer_id,
)
))
await self._receive_loop(ws)
except Exception as exc:
self._connected.clear()
if self._running:
log.debug("Gossip disconnected (%s); reconnecting in %ss", exc, self.reconnect_interval)
await asyncio.sleep(self.reconnect_interval)
async def _receive_loop(self, ws) -> None:
async for raw in ws:
try:
envelope = json.loads(raw)
except (json.JSONDecodeError, TypeError):
continue
msg_id = envelope.get("msg_id", "")
if msg_id in self._seen:
continue
self._seen.add(msg_id)
if len(self._seen) > 10_000:
# Trim seen set to avoid unbounded growth
self._seen = set(list(self._seen)[-5_000:])
topic = envelope.get("topic", "")
for handler in self._handlers.get(topic, []):
try:
handler(envelope)
except Exception:
log.debug("Gossip handler error for topic %s", topic, exc_info=True)
async def _send(self, envelope: dict) -> None:
if self._ws is not None:
try:
await self._ws.send(json.dumps(envelope))
except Exception as exc:
log.debug("Gossip send failed: %s", exc)

View File

@@ -0,0 +1,64 @@
"""Peer identity — stable peer_id and RSA keypair, persisted to disk."""
from __future__ import annotations
import hashlib
import json
import os
import stat
from pathlib import Path
_DEFAULT_IDENTITY_PATH = Path.home() / ".config" / "meshnet" / "identity.json"
def _generate_keypair() -> tuple[bytes, bytes]:
"""Return (private_key_pem, public_key_pem) for a new RSA-2048 keypair."""
from cryptography.hazmat.primitives.asymmetric import rsa
from cryptography.hazmat.primitives import serialization
key = rsa.generate_private_key(public_exponent=65537, key_size=2048)
priv_pem = key.private_bytes(
serialization.Encoding.PEM,
serialization.PrivateFormat.PKCS8,
serialization.NoEncryption(),
)
pub_pem = key.public_key().public_bytes(
serialization.Encoding.PEM,
serialization.PublicFormat.SubjectPublicKeyInfo,
)
return priv_pem, pub_pem
def _peer_id_from_pubkey(pub_pem: bytes) -> str:
return hashlib.sha256(pub_pem).hexdigest()[:16]
def load_or_create_identity(path: Path | None = None) -> dict:
"""Return identity dict with peer_id, private_key_pem, public_key_pem.
Creates and persists a new identity if none exists at path.
"""
p = path or _DEFAULT_IDENTITY_PATH
if p.exists():
try:
data = json.loads(p.read_text())
if "peer_id" in data and "public_key_pem" in data:
return data
except (json.JSONDecodeError, OSError):
pass
priv_pem, pub_pem = _generate_keypair()
identity = {
"peer_id": _peer_id_from_pubkey(pub_pem),
"private_key_pem": priv_pem.decode(),
"public_key_pem": pub_pem.decode(),
}
p.parent.mkdir(parents=True, exist_ok=True)
p.write_text(json.dumps(identity, indent=2))
try:
os.chmod(p, stat.S_IRUSR | stat.S_IWUSR)
except OSError:
pass
return identity

View File

@@ -0,0 +1,136 @@
"""mDNS peer discovery using zeroconf (optional dependency).
Falls back gracefully if zeroconf is not installed.
"""
from __future__ import annotations
import logging
import socket
import threading
from typing import Callable
log = logging.getLogger(__name__)
MDNS_SERVICE_TYPE = "_meshnet._tcp.local."
try:
from zeroconf import ServiceInfo, ServiceBrowser, Zeroconf # type: ignore[import]
_HAS_ZEROCONF = True
except ImportError:
_HAS_ZEROCONF = False
def _local_ip() -> str:
try:
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.connect(("8.8.8.8", 80))
ip = s.getsockname()[0]
s.close()
return ip
except OSError:
return "127.0.0.1"
class MdnsDiscovery:
"""Announce this node on mDNS and discover peers on the same LAN.
If `zeroconf` is not installed, all methods are no-ops.
Usage::
disc = MdnsDiscovery(
peer_id="abc123",
port=8001,
on_peer_found=lambda peer_id, addr: print("found", peer_id, addr),
)
disc.start()
...
disc.stop()
"""
def __init__(
self,
peer_id: str,
port: int,
on_peer_found: Callable[[str, str], None] | None = None,
on_peer_lost: Callable[[str], None] | None = None,
):
self.peer_id = peer_id
self.port = port
self.on_peer_found = on_peer_found
self.on_peer_lost = on_peer_lost
self._zc: "Zeroconf | None" = None # type: ignore[name-defined]
self._info: "ServiceInfo | None" = None # type: ignore[name-defined]
self._browser = None
def is_available(self) -> bool:
return _HAS_ZEROCONF
def start(self) -> None:
if not _HAS_ZEROCONF:
log.info("zeroconf not installed — mDNS discovery disabled")
return
try:
self._zc = Zeroconf()
local_ip = _local_ip()
self._info = ServiceInfo(
MDNS_SERVICE_TYPE,
f"{self.peer_id}.{MDNS_SERVICE_TYPE}",
addresses=[socket.inet_aton(local_ip)],
port=self.port,
properties={"peer_id": self.peer_id, "version": "1"},
)
self._zc.register_service(self._info)
if self.on_peer_found or self.on_peer_lost:
self._browser = ServiceBrowser(
self._zc, MDNS_SERVICE_TYPE, listener=_Listener(self)
)
log.info("mDNS announced: %s on %s:%d", self.peer_id, local_ip, self.port)
except Exception as exc:
log.warning("mDNS start failed: %s", exc)
def stop(self) -> None:
if not _HAS_ZEROCONF or self._zc is None:
return
try:
if self._info:
self._zc.unregister_service(self._info)
self._zc.close()
except Exception as exc:
log.debug("mDNS stop error: %s", exc)
self._zc = None
class _Listener:
"""Internal zeroconf service listener."""
def __init__(self, disc: MdnsDiscovery):
self._disc = disc
def add_service(self, zc, type_, name):
try:
info = zc.get_service_info(type_, name)
if info is None:
return
remote_peer_id = (info.properties or {}).get(b"peer_id", b"").decode()
if remote_peer_id == self._disc.peer_id:
return # ignore self
addr = f"http://{socket.inet_ntoa(info.addresses[0])}:{info.port}"
if self._disc.on_peer_found:
self._disc.on_peer_found(remote_peer_id, addr)
except Exception as exc:
log.debug("mDNS add_service error: %s", exc)
def remove_service(self, zc, type_, name):
try:
# name is like "peer_id._meshnet._tcp.local."
peer_id = name.split(".")[0]
if self._disc.on_peer_lost:
self._disc.on_peer_lost(peer_id)
except Exception as exc:
log.debug("mDNS remove_service error: %s", exc)
def update_service(self, zc, type_, name):
pass

View File

@@ -0,0 +1,114 @@
"""TLS certificate generation and fingerprint helpers for node-to-node comms."""
from __future__ import annotations
import datetime
import hashlib
import ipaddress
import json
import os
import socket
import ssl
import stat
from pathlib import Path
_CERT_PATH = Path.home() / ".config" / "meshnet" / "node_cert.pem"
_KEY_PATH = Path.home() / ".config" / "meshnet" / "node_key.pem"
def generate_self_signed_cert(
cert_path: Path | None = None,
key_path: Path | None = None,
common_name: str | None = None,
) -> tuple[Path, Path]:
"""Generate a self-signed RSA-2048 cert valid for 10 years.
Returns (cert_path, key_path). Skips generation if both files already exist.
"""
from cryptography import x509
from cryptography.hazmat.primitives import hashes, serialization
from cryptography.hazmat.primitives.asymmetric import rsa
from cryptography.x509.oid import NameOID
cert_p = cert_path or _CERT_PATH
key_p = key_path or _KEY_PATH
if cert_p.exists() and key_p.exists():
return cert_p, key_p
cert_p.parent.mkdir(parents=True, exist_ok=True)
key = rsa.generate_private_key(public_exponent=65537, key_size=2048)
cn = common_name or socket.getfqdn()
subject = issuer = x509.Name([
x509.NameAttribute(NameOID.COMMON_NAME, cn),
x509.NameAttribute(NameOID.ORGANIZATION_NAME, "meshnet-node"),
])
san_list: list = [x509.DNSName(cn)]
try:
san_list.append(x509.IPAddress(ipaddress.IPv4Address(socket.gethostbyname(cn))))
except (socket.gaierror, ValueError):
pass
san_list.append(x509.IPAddress(ipaddress.IPv4Address("127.0.0.1")))
cert = (
x509.CertificateBuilder()
.subject_name(subject)
.issuer_name(issuer)
.public_key(key.public_key())
.serial_number(x509.random_serial_number())
.not_valid_before(datetime.datetime.now(datetime.timezone.utc))
.not_valid_after(datetime.datetime.now(datetime.timezone.utc) + datetime.timedelta(days=365 * 10))
.add_extension(x509.SubjectAlternativeName(san_list), critical=False)
.sign(key, hashes.SHA256())
)
key_pem = key.private_bytes(
serialization.Encoding.PEM,
serialization.PrivateFormat.TraditionalOpenSSL,
serialization.NoEncryption(),
)
cert_pem = cert.public_bytes(serialization.Encoding.PEM)
key_p.write_bytes(key_pem)
cert_p.write_bytes(cert_pem)
try:
os.chmod(key_p, stat.S_IRUSR | stat.S_IWUSR)
except OSError:
pass
return cert_p, key_p
def cert_fingerprint(cert_path: Path | None = None) -> str:
"""Return sha256 fingerprint of the cert as 'sha256:<hex>'."""
from cryptography import x509
from cryptography.hazmat.primitives import hashes
p = cert_path or _CERT_PATH
cert = x509.load_pem_x509_certificate(p.read_bytes())
fp = cert.fingerprint(hashes.SHA256()).hex()
return f"sha256:{fp}"
def make_server_ssl_context(
cert_path: Path | None = None,
key_path: Path | None = None,
) -> ssl.SSLContext:
"""Return an ssl.SSLContext for a server using our self-signed cert."""
cert_p = cert_path or _CERT_PATH
key_p = key_path or _KEY_PATH
ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
ctx.load_cert_chain(certfile=str(cert_p), keyfile=str(key_p))
return ctx
def make_client_ssl_context(verify: bool = False) -> ssl.SSLContext:
"""Return a client SSLContext. verify=False for self-signed TOFU connections."""
ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
if not verify:
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
return ctx

View File

@@ -8,6 +8,14 @@ version = "0.1.0"
description = "Distributed Inference Network gossip and shard swarm" description = "Distributed Inference Network gossip and shard swarm"
requires-python = ">=3.10" requires-python = ">=3.10"
dependencies = [
"cryptography>=41",
"websockets>=13",
]
[project.optional-dependencies]
mdns = ["zeroconf>=0.131"]
[tool.setuptools.packages.find] [tool.setuptools.packages.find]
where = ["."] where = ["."]
include = ["meshnet_p2p*"] include = ["meshnet_p2p*"]

View File

@@ -0,0 +1,10 @@
{
"relays": [
{
"url": "ws://localhost:8765",
"cert_fingerprint": null,
"operator": "localhost-dev",
"note": "Local development relay — replace with team relay URL before production"
}
]
}

View File

@@ -0,0 +1,3 @@
"""meshnet-relay — NAT-traversal relay and gossip hub."""
__version__ = "0.1.0"

View File

@@ -0,0 +1,60 @@
"""meshnet-relay CLI entry point."""
from __future__ import annotations
import argparse
import logging
import sys
import time
from pathlib import Path
def main() -> None:
parser = argparse.ArgumentParser(
prog="meshnet-relay",
description="Meshnet NAT-traversal relay and gossip hub",
)
parser.add_argument("--host", default="0.0.0.0", help="Interface to bind")
parser.add_argument("--port", type=int, default=8765, help="WebSocket port")
parser.add_argument("--cert", metavar="PATH", help="TLS certificate (PEM)")
parser.add_argument("--key", metavar="PATH", help="TLS private key (PEM)")
parser.add_argument("--max-peers", type=int, default=500, help="Max concurrent peers")
parser.add_argument("--log-level", default="INFO", choices=["DEBUG", "INFO", "WARNING", "ERROR"])
args = parser.parse_args()
logging.basicConfig(
level=getattr(logging, args.log_level),
format="%(asctime)s %(levelname)-8s %(name)s %(message)s",
)
from .server import RelayServer
ssl_cert = Path(args.cert) if args.cert else None
ssl_key = Path(args.key) if args.key else None
server = RelayServer(
host=args.host,
port=args.port,
ssl_cert=ssl_cert,
ssl_key=ssl_key,
max_peers=args.max_peers,
)
port = server.start()
scheme = "wss" if ssl_cert else "ws"
print(f"meshnet-relay listening on {scheme}://{args.host}:{port}", flush=True)
print(" /ws gossip PubSub", flush=True)
print(" /relay/<id> circuit relay to peer", flush=True)
print(" /health health check", flush=True)
print(" /v1/peers peer list", flush=True)
print("Press Ctrl-C to stop.", flush=True)
try:
while True:
time.sleep(1)
except KeyboardInterrupt:
print("\nStopping relay…", flush=True)
server.stop()
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,60 @@
"""In-memory registry of connected gossip peers."""
from __future__ import annotations
import threading
import time
from dataclasses import dataclass, field
from typing import Any
@dataclass
class PeerEntry:
peer_id: str
addr: str
ws: Any # websockets.WebSocketServerProtocol
connected_at: float = field(default_factory=time.monotonic)
last_seen: float = field(default_factory=time.monotonic)
class PeerRegistry:
def __init__(self):
self._peers: dict[str, PeerEntry] = {}
self._lock = threading.Lock()
def register(self, peer_id: str, addr: str, ws) -> None:
with self._lock:
self._peers[peer_id] = PeerEntry(peer_id=peer_id, addr=addr, ws=ws)
def unregister(self, peer_id: str) -> None:
with self._lock:
self._peers.pop(peer_id, None)
def touch(self, peer_id: str) -> None:
with self._lock:
if peer_id in self._peers:
self._peers[peer_id].last_seen = time.monotonic()
def get(self, peer_id: str) -> PeerEntry | None:
with self._lock:
return self._peers.get(peer_id)
def all_except(self, peer_id: str) -> list[PeerEntry]:
with self._lock:
return [e for pid, e in self._peers.items() if pid != peer_id]
def list_peers(self) -> list[dict]:
with self._lock:
return [
{
"peer_id": e.peer_id,
"addr": e.addr,
"connected_at": e.connected_at,
"last_seen": e.last_seen,
}
for e in self._peers.values()
]
def __len__(self) -> int:
with self._lock:
return len(self._peers)

View File

@@ -0,0 +1,224 @@
"""Relay server — WebSocket gossip hub + circuit relay proxy.
HTTP API (served via asyncio-based handler on same port):
GET /health → {"status": "ok", "peers": N}
GET /v1/peers → [{peer_id, addr, last_seen}]
POST /v1/gossip → accept a gossip envelope, fan out to connected peers
WebSocket endpoints:
ws[s]://host:port/ws → gossip PubSub connection
ws[s]://host:port/relay/{peer_id} → circuit relay to that peer
"""
from __future__ import annotations
import asyncio
import json
import logging
import threading
from pathlib import Path
from .peer_registry import PeerRegistry
log = logging.getLogger(__name__)
class RelayServer:
"""Async WebSocket relay server that runs in a background thread.
Usage::
server = RelayServer(host="0.0.0.0", port=8765)
port = server.start() # returns actual port
...
server.stop()
"""
def __init__(
self,
host: str = "0.0.0.0",
port: int = 8765,
ssl_cert: Path | None = None,
ssl_key: Path | None = None,
max_peers: int = 500,
):
self.host = host
self.port = port
self.ssl_cert = ssl_cert
self.ssl_key = ssl_key
self.max_peers = max_peers
self._registry = PeerRegistry()
self._loop: asyncio.AbstractEventLoop | None = None
self._thread: threading.Thread | None = None
self._server = None
self._actual_port: int = port
self._ready = threading.Event()
self._running = False
self._stop_event: asyncio.Event | None = None
@property
def registry(self) -> PeerRegistry:
return self._registry
def start(self) -> int:
"""Start server in background thread. Returns actual bound port."""
self._running = True
self._loop = asyncio.new_event_loop()
self._thread = threading.Thread(target=self._run, daemon=True, name="relay")
self._thread.start()
self._ready.wait(timeout=5)
return self._actual_port
def stop(self) -> None:
self._running = False
if self._loop and self._stop_event is not None:
self._loop.call_soon_threadsafe(self._stop_event.set)
if self._thread:
self._thread.join(timeout=3.0)
def _run(self) -> None:
asyncio.set_event_loop(self._loop)
self._loop.run_until_complete(self._serve())
async def _serve(self) -> None:
import websockets # type: ignore[import]
import websockets.server # type: ignore[import]
ssl_ctx = None
if self.ssl_cert and self.ssl_key:
import ssl
ssl_ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
ssl_ctx.load_cert_chain(str(self.ssl_cert), str(self.ssl_key))
server = await websockets.serve(
self._handle_connection,
self.host,
self.port,
ssl=ssl_ctx,
)
# Record actual port after bind
for sock in server.sockets or []:
self._actual_port = sock.getsockname()[1]
break
self._stop_event = asyncio.Event()
self._server = server
self._ready.set()
log.info("Relay listening on %s:%d", self.host, self._actual_port)
await self._stop_event.wait()
server.close()
await server.wait_closed()
async def _handle_connection(self, ws) -> None:
"""Dispatch incoming WebSocket to gossip hub or circuit relay."""
try:
path = ws.request.path
except AttributeError:
path = getattr(ws, "path", "/ws")
if path.startswith("/relay/"):
peer_id = path[len("/relay/"):]
await self._handle_circuit_relay(ws, peer_id)
elif path == "/health":
await ws.send(json.dumps({"status": "ok", "peers": len(self._registry)}))
await ws.close()
elif path == "/v1/peers":
await ws.send(json.dumps(self._registry.list_peers()))
await ws.close()
else:
await self._handle_gossip(ws)
async def _handle_gossip(self, ws) -> None:
"""Accept a gossip peer connection, register it, and fan out messages."""
peer_id: str | None = None
peer_addr: str = ""
try:
async for raw in ws:
try:
envelope = json.loads(raw)
except (json.JSONDecodeError, TypeError):
continue
topic = envelope.get("topic", "")
from_peer = envelope.get("from_peer", "")
# Handle peer registration message
if topic == "peer-register":
payload = envelope.get("payload", {})
peer_id = payload.get("peer_id") or from_peer
peer_addr = payload.get("addr", "")
if len(self._registry) >= self.max_peers:
await ws.close(1008, "relay at capacity")
return
self._registry.register(peer_id, peer_addr, ws)
log.debug("Peer registered: %s", peer_id)
# Send current peer list back
await ws.send(json.dumps({
"topic": "peer-list",
"version": 1,
"from_peer": "relay",
"payload": {"peers": self._registry.list_peers()},
}))
continue
# Fan out to all other registered peers
if peer_id:
self._registry.touch(peer_id)
fan_out_peers = self._registry.all_except(peer_id or "")
await _broadcast(raw, fan_out_peers)
except Exception as exc:
log.debug("Gossip connection error: %s", exc)
finally:
if peer_id:
self._registry.unregister(peer_id)
log.debug("Peer unregistered: %s", peer_id)
async def _handle_circuit_relay(self, ws_requester, target_peer_id: str) -> None:
"""Proxy WebSocket traffic between ws_requester and target_peer_id's ws."""
target = self._registry.get(target_peer_id)
if target is None:
try:
await ws_requester.send(json.dumps({
"error": f"peer {target_peer_id!r} not connected to relay"
}))
await ws_requester.close(1011, "target peer not found")
except Exception:
pass
return
log.debug("Circuit relay: ??? → %s", target_peer_id)
async def pipe(src, dst) -> None:
try:
async for msg in src:
await dst.send(msg)
except Exception:
pass
await asyncio.gather(
pipe(ws_requester, target.ws),
pipe(target.ws, ws_requester),
return_exceptions=True,
)
async def _broadcast(raw: str | bytes, peers: list) -> None:
"""Send raw message to all peers; ignore individual send failures."""
if not peers:
return
import asyncio
await asyncio.gather(
*[_safe_send(p.ws, raw) for p in peers],
return_exceptions=True,
)
async def _safe_send(ws, msg) -> None:
try:
await ws.send(msg)
except Exception:
pass

View File

@@ -0,0 +1,20 @@
[build-system]
requires = ["setuptools>=64"]
build-backend = "setuptools.build_meta"
[project]
name = "meshnet-relay"
version = "0.1.0"
description = "Distributed Inference Network NAT-traversal relay and gossip hub"
requires-python = ">=3.10"
dependencies = [
"websockets>=13",
]
[project.scripts]
meshnet-relay = "meshnet_relay.cli:main"
[tool.setuptools.packages.find]
where = ["."]
include = ["meshnet_relay*"]

View File

@@ -31,28 +31,55 @@ from typing import Any
DEFAULT_MODEL_PRESETS: dict[str, dict] = { DEFAULT_MODEL_PRESETS: dict[str, dict] = {
"stub-model": {"layers_start": 0, "layers_end": 31}, "stub-model": {
"layers_start": 0,
"layers_end": 31,
"bytes_per_layer": {"bfloat16": 30 * 1024 * 1024},
},
"openai-community/gpt2": {
"layers_start": 0,
"layers_end": 11,
"bytes_per_layer": {"bfloat16": 30 * 1024 * 1024, "int8": 15 * 1024 * 1024, "nf4": 8 * 1024 * 1024},
},
} }
DEFAULT_VRAM_BYTES = 8 * 1024 * 1024 * 1024
DEFAULT_RAM_BYTES = 16 * 1024 * 1024 * 1024
DEFAULT_QUANTIZATIONS = ["bfloat16"]
DEFAULT_BENCHMARK_TOKENS_PER_SEC = 1.0
class _NodeEntry: class _NodeEntry:
__slots__ = ( __slots__ = (
"node_id", "endpoint", "shard_start", "shard_end", "node_id", "endpoint", "shard_start", "shard_end",
"model", "shard_checksum", "hardware_profile", "wallet_address", "model", "shard_checksum", "hardware_profile", "wallet_address",
"score", "last_heartbeat", "score", "vram_bytes", "ram_bytes", "quantizations",
"benchmark_tokens_per_sec", "quantization", "managed_assignment",
"pending_directives", "last_heartbeat", "tracker_mode",
"relay_addr", "cert_fingerprint", "peer_id",
) )
def __init__( def __init__(
self, self,
node_id: str, node_id: str,
endpoint: str, endpoint: str,
shard_start: int, shard_start: int | None,
shard_end: int, shard_end: int | None,
model: str | None, model: str | None,
shard_checksum: str | None, shard_checksum: str | None,
hardware_profile: dict, hardware_profile: dict,
wallet_address: str | None, wallet_address: str | None,
score: float, score: float,
vram_bytes: int = DEFAULT_VRAM_BYTES,
ram_bytes: int = DEFAULT_RAM_BYTES,
quantizations: list[str] | None = None,
benchmark_tokens_per_sec: float = DEFAULT_BENCHMARK_TOKENS_PER_SEC,
quantization: str | None = None,
managed_assignment: bool = False,
tracker_mode: bool = False,
relay_addr: str | None = None,
cert_fingerprint: str | None = None,
peer_id: str | None = None,
) -> None: ) -> None:
self.node_id = node_id self.node_id = node_id
self.endpoint = endpoint self.endpoint = endpoint
@@ -63,6 +90,17 @@ class _NodeEntry:
self.hardware_profile = hardware_profile self.hardware_profile = hardware_profile
self.wallet_address = wallet_address self.wallet_address = wallet_address
self.score = score self.score = score
self.vram_bytes = vram_bytes
self.ram_bytes = ram_bytes
self.quantizations = quantizations or list(DEFAULT_QUANTIZATIONS)
self.benchmark_tokens_per_sec = benchmark_tokens_per_sec
self.quantization = quantization
self.managed_assignment = managed_assignment
self.tracker_mode = tracker_mode
self.relay_addr = relay_addr
self.cert_fingerprint = cert_fingerprint
self.peer_id = peer_id
self.pending_directives: list[dict] = []
self.last_heartbeat: float = time.monotonic() self.last_heartbeat: float = time.monotonic()
@@ -72,7 +110,10 @@ def _select_route(
required_end: int, required_end: int,
) -> tuple[list[_NodeEntry], str]: ) -> tuple[list[_NodeEntry], str]:
"""Greedy interval-cover. Returns (ordered route, error_message).""" """Greedy interval-cover. Returns (ordered route, error_message)."""
candidates = sorted(nodes, key=lambda n: (n.shard_start, -n.shard_end)) candidates = sorted(
[node for node in nodes if node.shard_start is not None and node.shard_end is not None],
key=lambda n: (n.shard_start, -n.shard_end), # type: ignore[operator]
)
route: list[_NodeEntry] = [] route: list[_NodeEntry] = []
covered_up_to = required_start - 1 covered_up_to = required_start - 1
@@ -105,6 +146,8 @@ def _coverage_percentage(
( (
(max(required_start, node.shard_start), min(required_end, node.shard_end)) (max(required_start, node.shard_start), min(required_end, node.shard_end))
for node in nodes for node in nodes
if node.shard_start is not None
and node.shard_end is not None
if node.shard_end >= required_start and node.shard_start <= required_end if node.shard_end >= required_start and node.shard_start <= required_end
), ),
key=lambda interval: interval[0], key=lambda interval: interval[0],
@@ -122,6 +165,197 @@ def _coverage_percentage(
return round((covered / required_layers) * 100, 2) return round((covered / required_layers) * 100, 2)
def _preset_layer_bounds(preset: dict) -> tuple[int, int]:
start = int(preset.get("layers_start", 0))
if "layers_end" in preset:
return start, int(preset["layers_end"])
return start, start + int(preset["total_layers"]) - 1
def _preset_bytes_per_layer(preset: dict) -> dict[str, int]:
raw = preset.get("bytes_per_layer", preset.get("bytes_per_layer_at_quant", {}))
if isinstance(raw, dict) and raw:
return {str(quant): int(value) for quant, value in raw.items()}
return {"bfloat16": 30 * 1024 * 1024}
def _node_quantization(node: _NodeEntry, preset: dict) -> str:
bytes_per_layer = _preset_bytes_per_layer(preset)
if node.quantization in bytes_per_layer:
return node.quantization
for quantization in node.quantizations:
if quantization in bytes_per_layer:
return quantization
return next(iter(bytes_per_layer))
def _node_layer_capacity(node: _NodeEntry, preset: dict) -> int:
bytes_per_layer = _preset_bytes_per_layer(preset)
quantization = _node_quantization(node, preset)
layer_bytes = bytes_per_layer[quantization]
if layer_bytes <= 0:
return 0
return int((node.vram_bytes * 0.8) // layer_bytes)
def _coverage_map(
nodes: list[_NodeEntry],
required_start: int,
required_end: int,
) -> list[dict]:
layer_counts = []
for layer in range(required_start, required_end + 1):
count = 0
for node in nodes:
if node.shard_start is None or node.shard_end is None:
continue
if node.shard_start <= layer <= node.shard_end:
count += 1
layer_counts.append((layer, count))
coverage: list[dict] = []
for layer, count in layer_counts:
if coverage and coverage[-1]["node_count"] == count and coverage[-1]["end_layer"] == layer - 1:
coverage[-1]["end_layer"] = layer
else:
coverage.append({"start_layer": layer, "end_layer": layer, "node_count": count})
return coverage
def _coverage_gaps(coverage: list[dict]) -> list[tuple[int, int]]:
return [
(segment["start_layer"], segment["end_layer"])
for segment in coverage
if segment["node_count"] == 0
]
def _load_directive(node: _NodeEntry, model: str, start: int, end: int, quantization: str) -> dict:
return {
"action": "LOAD_SHARD",
"model": model,
"start_layer": start,
"end_layer": end,
"shard_start": start,
"shard_end": end,
"quantization": quantization,
}
def _drop_directive(node: _NodeEntry, model: str, start: int, end: int, quantization: str) -> dict:
return {
"action": "DROP_SHARD",
"model": model,
"start_layer": start,
"end_layer": end,
"shard_start": start,
"shard_end": end,
"quantization": quantization,
}
def _purge_expired_nodes_locked(server: "_TrackerHTTPServer") -> list[str]:
now = time.monotonic()
expired_ids = [
node_id for node_id, entry in server.registry.items()
if (now - entry.last_heartbeat) > server.heartbeat_timeout
]
for node_id in expired_ids:
del server.registry[node_id]
if expired_ids:
_rebalance_all_locked(server)
return expired_ids
def _rebalance_model_locked(server: "_TrackerHTTPServer", model: str) -> None:
preset = server.model_presets.get(model)
if preset is None:
return
required_start, required_end = _preset_layer_bounds(preset)
total_layers = required_end - required_start + 1
model_nodes = [node for node in server.registry.values() if node.model == model]
managed_nodes = [node for node in model_nodes if node.managed_assignment]
if not managed_nodes:
return
previous_ranges = {
node.node_id: (node.shard_start, node.shard_end, node.quantization)
for node in managed_nodes
}
for node in managed_nodes:
node.shard_start = None
node.shard_end = None
managed_nodes.sort(
key=lambda node: (
-node.benchmark_tokens_per_sec,
-_node_layer_capacity(node, preset),
node.node_id,
)
)
base_nodes = [node for node in model_nodes if not node.managed_assignment]
coverage = _coverage_map(base_nodes, required_start, required_end)
gaps = _coverage_gaps(coverage)
if not gaps:
gaps = [(required_start, required_end)]
eligible_nodes = [
node for node in managed_nodes
if _node_layer_capacity(node, preset) > 0
]
node_index = 0
for gap_start, gap_end in gaps:
cursor = gap_start
while cursor <= gap_end and node_index < len(eligible_nodes):
node = eligible_nodes[node_index]
remaining_layers = gap_end - cursor + 1
remaining_nodes_after = len(eligible_nodes) - node_index - 1
capacity = min(
_node_layer_capacity(node, preset),
total_layers,
max(1, remaining_layers - remaining_nodes_after),
)
if capacity <= 0:
node_index += 1
continue
quantization = _node_quantization(node, preset)
node.quantization = quantization
node.shard_start = cursor
node.shard_end = min(gap_end, cursor + capacity - 1)
cursor = node.shard_end + 1
node_index += 1
for node in managed_nodes:
previous_start, previous_end, previous_quantization = previous_ranges[node.node_id]
current_range = (node.shard_start, node.shard_end, node.quantization)
if node.shard_start is None or node.shard_end is None or current_range == previous_ranges[node.node_id]:
continue
if previous_start is not None and previous_end is not None:
node.pending_directives.append(
_drop_directive(
node,
model,
previous_start,
previous_end,
previous_quantization or _node_quantization(node, preset),
)
)
node.pending_directives.append(
_load_directive(
node,
model,
node.shard_start,
node.shard_end,
node.quantization or _node_quantization(node, preset),
)
)
def _rebalance_all_locked(server: "_TrackerHTTPServer") -> None:
for model in list(server.model_presets):
_rebalance_model_locked(server, model)
def _registration_ban_error(contracts: Any | None, wallet_address: str | None) -> str | None: def _registration_ban_error(contracts: Any | None, wallet_address: str | None) -> str | None:
if contracts is None or not wallet_address: if contracts is None or not wallet_address:
return None return None
@@ -177,13 +411,7 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
def _purge_expired_nodes(self) -> None: def _purge_expired_nodes(self) -> None:
server: _TrackerHTTPServer = self.server # type: ignore[assignment] server: _TrackerHTTPServer = self.server # type: ignore[assignment]
now = time.monotonic() _purge_expired_nodes_locked(server)
expired_ids = [
node_id for node_id, entry in server.registry.items()
if (now - entry.last_heartbeat) > server.heartbeat_timeout
]
for node_id in expired_ids:
del server.registry[node_id]
def do_POST(self): def do_POST(self):
if self.path == "/v1/nodes/register": if self.path == "/v1/nodes/register":
@@ -207,6 +435,12 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
self._handle_assign(parsed) self._handle_assign(parsed)
elif parsed.path == "/v1/models": elif parsed.path == "/v1/models":
self._handle_models() self._handle_models()
elif parsed.path.startswith("/v1/coverage/"):
model = urllib.parse.unquote(parsed.path.removeprefix("/v1/coverage/"))
self._handle_coverage(model)
elif parsed.path.startswith("/v1/tracker-nodes/"):
model = urllib.parse.unquote(parsed.path.removeprefix("/v1/tracker-nodes/"))
self._handle_tracker_nodes(model)
else: else:
self.send_response(404) self.send_response(404)
self.end_headers() self.end_headers()
@@ -225,10 +459,13 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
data = [] data = []
for name, preset in server.model_presets.items(): for name, preset in server.model_presets.items():
model_nodes = [node for node in alive if node.model == name] model_nodes = [node for node in alive if node.model == name]
if not model_nodes:
continue
required_start, required_end = _preset_layer_bounds(preset)
coverage = _coverage_percentage( coverage = _coverage_percentage(
model_nodes, model_nodes,
preset["layers_start"], required_start,
preset["layers_end"], required_end,
) )
data.append({ data.append({
"id": name, "id": name,
@@ -239,6 +476,58 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
}) })
self._send_json(200, {"object": "list", "data": data}) self._send_json(200, {"object": "list", "data": data})
def _handle_coverage(self, model: str):
server: _TrackerHTTPServer = self.server # type: ignore[assignment]
preset = server.model_presets.get(model)
if preset is None:
self._send_json(404, {"error": f"unknown model preset: {model!r}"})
return
required_start, required_end = _preset_layer_bounds(preset)
with server.lock:
self._purge_expired_nodes()
alive = [node for node in server.registry.values() if node.model == model]
if server.contracts is not None:
alive = [
node for node in alive
if not node.wallet_address or not server.contracts.registry.get_wallet(node.wallet_address).banned
]
coverage = _coverage_map(alive, required_start, required_end)
self._send_json(200, {"model": model, "coverage": coverage})
def _handle_tracker_nodes(self, model: str):
"""Return nodes registered with tracker_mode=True whose shard starts at layer 0."""
server: _TrackerHTTPServer = self.server # type: ignore[assignment]
preset = server.model_presets.get(model)
if preset is None:
self._send_json(404, {"error": f"unknown model preset: {model!r}"})
return
required_start, _ = _preset_layer_bounds(preset)
with server.lock:
self._purge_expired_nodes()
alive = [node for node in server.registry.values() if node.model == model]
if server.contracts is not None:
alive = [
node for node in alive
if not node.wallet_address or not server.contracts.registry.get_wallet(node.wallet_address).banned
]
tracker_nodes = [
node for node in alive
if node.shard_start is not None
and node.shard_start == required_start
and node.tracker_mode
]
self._send_json(200, {
"model": model,
"tracker_nodes": [
{
"node_id": node.node_id,
"endpoint": node.endpoint,
"benchmark_tokens_per_sec": node.benchmark_tokens_per_sec,
}
for node in tracker_nodes
],
})
def _handle_register(self): def _handle_register(self):
server: _TrackerHTTPServer = self.server # type: ignore[assignment] server: _TrackerHTTPServer = self.server # type: ignore[assignment]
body = self._read_json_body() body = self._read_json_body()
@@ -254,15 +543,26 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
self._send_json(400, {"error": "endpoint must be an http(s) URL"}) self._send_json(400, {"error": "endpoint must be an http(s) URL"})
return return
shard_start: int | None
shard_end: int | None
explicit_shard = "shard_start" in body or "shard_end" in body
if explicit_shard:
try:
shard_start = int(body["shard_start"])
shard_end = int(body["shard_end"])
except (KeyError, TypeError, ValueError):
self._send_json(400, {"error": "shard_start and shard_end must be numeric"})
return
if shard_start < 0 or shard_end < 0 or shard_start > shard_end:
self._send_json(400, {"error": "shard range must be non-negative and ordered"})
return
else:
shard_start = None
shard_end = None
try: try:
shard_start = int(body["shard_start"])
shard_end = int(body["shard_end"])
score = float(body.get("score", 1.0)) score = float(body.get("score", 1.0))
except (KeyError, TypeError, ValueError): except (TypeError, ValueError):
self._send_json(400, {"error": "shard_start, shard_end, and score must be numeric"}) self._send_json(400, {"error": "score must be numeric"})
return
if shard_start < 0 or shard_end < 0 or shard_start > shard_end:
self._send_json(400, {"error": "shard range must be non-negative and ordered"})
return return
hardware_profile = body.get("hardware_profile", {}) hardware_profile = body.get("hardware_profile", {})
@@ -279,6 +579,31 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
if shard_checksum is not None and not isinstance(shard_checksum, str): if shard_checksum is not None and not isinstance(shard_checksum, str):
self._send_json(400, {"error": "shard_checksum must be a string"}) self._send_json(400, {"error": "shard_checksum must be a string"})
return return
try:
vram_bytes = int(body.get("vram_bytes", DEFAULT_VRAM_BYTES))
ram_bytes = int(body.get("ram_bytes", DEFAULT_RAM_BYTES))
benchmark_tokens_per_sec = float(
body.get("benchmark_tokens_per_sec", DEFAULT_BENCHMARK_TOKENS_PER_SEC)
)
except (TypeError, ValueError):
self._send_json(400, {"error": "vram_bytes, ram_bytes, and benchmark_tokens_per_sec must be numeric"})
return
if vram_bytes < 0 or ram_bytes < 0 or benchmark_tokens_per_sec <= 0:
self._send_json(400, {"error": "capability values must be positive"})
return
quantizations_body = body.get("quantizations", DEFAULT_QUANTIZATIONS)
if not (
isinstance(quantizations_body, list)
and quantizations_body
and all(isinstance(item, str) and item for item in quantizations_body)
):
self._send_json(400, {"error": "quantizations must be a non-empty string array"})
return
quantizations = list(quantizations_body)
quantization = body.get("quantization")
if quantization is not None and not isinstance(quantization, str):
self._send_json(400, {"error": "quantization must be a string"})
return
wallet_address = body.get("wallet_address") wallet_address = body.get("wallet_address")
if wallet_address is not None and not isinstance(wallet_address, str): if wallet_address is not None and not isinstance(wallet_address, str):
self._send_json(400, {"error": "wallet_address must be a string"}) self._send_json(400, {"error": "wallet_address must be a string"})
@@ -288,6 +613,11 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
self._send_json(403, {"error": ban_error}) self._send_json(403, {"error": ban_error})
return return
tracker_mode = bool(body.get("tracker_mode", False))
relay_addr = body.get("relay_addr") or None
cert_fingerprint = body.get("cert_fingerprint") or None
peer_id = body.get("peer_id") or None
node_id = str(uuid.uuid4()) node_id = str(uuid.uuid4())
entry = _NodeEntry( entry = _NodeEntry(
node_id=node_id, node_id=node_id,
@@ -299,15 +629,40 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
hardware_profile=hardware_profile, hardware_profile=hardware_profile,
wallet_address=wallet_address, wallet_address=wallet_address,
score=score, score=score,
vram_bytes=vram_bytes,
ram_bytes=ram_bytes,
quantizations=quantizations,
benchmark_tokens_per_sec=benchmark_tokens_per_sec,
quantization=quantization,
managed_assignment=not explicit_shard,
tracker_mode=tracker_mode,
relay_addr=relay_addr,
cert_fingerprint=cert_fingerprint,
peer_id=peer_id,
) )
with server.lock: with server.lock:
self._purge_expired_nodes() self._purge_expired_nodes()
server.registry[node_id] = entry server.registry[node_id] = entry
if entry.managed_assignment:
_rebalance_model_locked(server, model)
assignment_directive = entry.pending_directives[-1] if entry.pending_directives else None
if assignment_directive is not None:
entry.pending_directives.clear()
self._send_json(200, {"node_id": node_id}) payload = {"node_id": node_id}
if assignment_directive is not None:
payload["directive"] = assignment_directive
self._send_json(200, payload)
def _handle_heartbeat(self, node_id: str): def _handle_heartbeat(self, node_id: str):
server: _TrackerHTTPServer = self.server # type: ignore[assignment] server: _TrackerHTTPServer = self.server # type: ignore[assignment]
body: dict = {}
content_length = int(self.headers.get("Content-Length", 0))
if content_length > 0:
try:
body = json.loads(self.rfile.read(content_length))
except Exception:
pass
with server.lock: with server.lock:
self._purge_expired_nodes() self._purge_expired_nodes()
entry = server.registry.get(node_id) entry = server.registry.get(node_id)
@@ -315,7 +670,19 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
self._send_json(404, {"error": "node not found"}) self._send_json(404, {"error": "node not found"})
return return
entry.last_heartbeat = time.monotonic() entry.last_heartbeat = time.monotonic()
self._send_json(200, {}) if body.get("relay_addr"):
entry.relay_addr = body["relay_addr"]
if body.get("cert_fingerprint"):
entry.cert_fingerprint = body["cert_fingerprint"]
if body.get("peer_id"):
entry.peer_id = body["peer_id"]
_rebalance_model_locked(server, entry.model or "stub-model")
directives = list(entry.pending_directives)
entry.pending_directives.clear()
if directives:
self._send_json(200, {"directives": directives})
else:
self._send_json(200, {})
def _handle_assign(self, parsed: urllib.parse.ParseResult): def _handle_assign(self, parsed: urllib.parse.ParseResult):
"""Return an optimal shard assignment for a node given its hardware profile. """Return an optimal shard assignment for a node given its hardware profile.
@@ -346,8 +713,7 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
self._send_json(404, {"error": f"unknown model preset: {model!r}"}) self._send_json(404, {"error": f"unknown model preset: {model!r}"})
return return
required_start: int = preset["layers_start"] required_start, required_end = _preset_layer_bounds(preset)
required_end: int = preset["layers_end"]
with server.lock: with server.lock:
self._purge_expired_nodes() self._purge_expired_nodes()
@@ -369,7 +735,11 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
# Collect covered intervals sorted by start layer. # Collect covered intervals sorted by start layer.
covered = sorted( covered = sorted(
[(n.shard_start, n.shard_end) for n in alive], [
(n.shard_start, n.shard_end)
for n in alive
if n.shard_start is not None and n.shard_end is not None
],
key=lambda t: t[0], key=lambda t: t[0],
) )
@@ -421,8 +791,7 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
self._send_json(404, {"error": f"unknown model preset: {model!r}"}) self._send_json(404, {"error": f"unknown model preset: {model!r}"})
return return
required_start: int = preset["layers_start"] required_start, required_end = _preset_layer_bounds(preset)
required_end: int = preset["layers_end"]
with server.lock: with server.lock:
self._purge_expired_nodes() self._purge_expired_nodes()
@@ -477,8 +846,7 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
self._send_json(404, {"error": f"unknown model preset: {model!r}"}) self._send_json(404, {"error": f"unknown model preset: {model!r}"})
return return
required_start: int = preset["layers_start"] required_start, required_end = _preset_layer_bounds(preset)
required_end: int = preset["layers_end"]
with server.lock: with server.lock:
self._purge_expired_nodes() self._purge_expired_nodes()
@@ -530,6 +898,7 @@ class TrackerServer:
host: str = "127.0.0.1", host: str = "127.0.0.1",
port: int = 0, port: int = 0,
heartbeat_timeout: float = 30.0, heartbeat_timeout: float = 30.0,
rebalance_interval: float = 30.0,
model_presets: dict | None = None, model_presets: dict | None = None,
contracts: Any | None = None, contracts: Any | None = None,
minimum_stake: int = 0, minimum_stake: int = 0,
@@ -537,6 +906,7 @@ class TrackerServer:
self._host = host self._host = host
self._requested_port = port self._requested_port = port
self._heartbeat_timeout = heartbeat_timeout self._heartbeat_timeout = heartbeat_timeout
self._rebalance_interval = rebalance_interval
self._model_presets: dict = ( self._model_presets: dict = (
model_presets if model_presets is not None else dict(DEFAULT_MODEL_PRESETS) model_presets if model_presets is not None else dict(DEFAULT_MODEL_PRESETS)
) )
@@ -546,6 +916,8 @@ class TrackerServer:
self._lock = threading.Lock() self._lock = threading.Lock()
self._server: _TrackerHTTPServer | None = None self._server: _TrackerHTTPServer | None = None
self._thread: threading.Thread | None = None self._thread: threading.Thread | None = None
self._rebalance_stop = threading.Event()
self._rebalance_thread: threading.Thread | None = None
self.port: int | None = None self.port: int | None = None
def start(self) -> int: def start(self) -> int:
@@ -562,17 +934,33 @@ class TrackerServer:
self._minimum_stake, self._minimum_stake,
) )
self.port = self._server.server_address[1] self.port = self._server.server_address[1]
self._rebalance_stop.clear()
self._thread = threading.Thread(target=self._server.serve_forever, daemon=True) self._thread = threading.Thread(target=self._server.serve_forever, daemon=True)
self._thread.start() self._thread.start()
self._rebalance_thread = threading.Thread(target=self._rebalance_loop, daemon=True)
self._rebalance_thread.start()
return self.port return self.port
def _rebalance_loop(self) -> None:
while not self._rebalance_stop.wait(self._rebalance_interval):
server = self._server
if server is None:
return
with self._lock:
_purge_expired_nodes_locked(server)
_rebalance_all_locked(server)
def stop(self) -> None: def stop(self) -> None:
if self._server is None: if self._server is None:
return return
self._rebalance_stop.set()
self._server.shutdown() self._server.shutdown()
self._server.server_close() self._server.server_close()
if self._thread is not None: if self._thread is not None:
self._thread.join(timeout=1) self._thread.join(timeout=1)
if self._rebalance_thread is not None:
self._rebalance_thread.join(timeout=1)
self._server = None self._server = None
self._thread = None self._thread = None
self._rebalance_thread = None
self.port = None self.port = None

View File

@@ -7,8 +7,11 @@ Examples:
python scripts/ralph_progress.py watch --interval 5 python scripts/ralph_progress.py watch --interval 5
python scripts/ralph_progress.py run-next --interval 10 python scripts/ralph_progress.py run-next --interval 10
python scripts/ralph_progress.py auto --max-tasks 3 --interval 10 python scripts/ralph_progress.py auto --max-tasks 3 --interval 10
python scripts/ralph_progress.py auto --parallel 2
python scripts/ralph_progress.py review --output .ralph-tui/reviews/task-doc-code-review.md python scripts/ralph_progress.py review --output .ralph-tui/reviews/task-doc-code-review.md
python scripts/ralph_progress.py review --run-ralph python scripts/ralph_progress.py review --run-ralph
python scripts/ralph_progress.py set-agent --agent claude
python scripts/ralph_progress.py list-parallel
""" """
from __future__ import annotations from __future__ import annotations
@@ -22,6 +25,7 @@ import shutil
import subprocess import subprocess
import sys import sys
import tempfile import tempfile
import threading
import time import time
from pathlib import Path from pathlib import Path
from textwrap import shorten from textwrap import shorten
@@ -32,6 +36,129 @@ REPO_ROOT = Path(__file__).resolve().parents[1]
DEFAULT_PRD = REPO_ROOT / ".scratch/distributed-inference-network/prd.json" DEFAULT_PRD = REPO_ROOT / ".scratch/distributed-inference-network/prd.json"
DEFAULT_AGENT = "codex" DEFAULT_AGENT = "codex"
DEFAULT_INTERVAL = 10.0 DEFAULT_INTERVAL = 10.0
AGENT_CONFIG_PATH = REPO_ROOT / ".ralph-tui" / "agent-config.json"
# ---------------------------------------------------------------------------
# Status vocabulary helpers
# ---------------------------------------------------------------------------
_DONE_STATUSES = {"done"}
_ATTENTION_STATUSES = {"to-revise", "needs-review"}
_ACTIVE_STATUSES = {"in-progress"}
_DESIGN_STATUSES = {"in-design"}
def _is_done(story: dict) -> bool:
status = story.get("status")
if status:
return status in _DONE_STATUSES
return bool(story.get("passes")) # backward compat
def _needs_attention(story: dict) -> bool:
return story.get("status") in _ATTENTION_STATUSES
def _is_active(story: dict) -> bool:
return story.get("status") in _ACTIVE_STATUSES
def _is_in_design(story: dict) -> bool:
return story.get("status") in _DESIGN_STATUSES
# ---------------------------------------------------------------------------
# Agent config helpers
# ---------------------------------------------------------------------------
def _load_agent_config() -> dict:
try:
return json.loads(AGENT_CONFIG_PATH.read_text())
except (FileNotFoundError, json.JSONDecodeError):
return {}
def _save_agent_config(agent: str, model: str | None = None) -> None:
AGENT_CONFIG_PATH.parent.mkdir(parents=True, exist_ok=True)
config: dict[str, Any] = {"agent": agent, "updatedAt": dt.datetime.now().isoformat()}
if model:
config["model"] = model
AGENT_CONFIG_PATH.write_text(json.dumps(config, indent=2))
def _default_agent() -> str:
cfg = _load_agent_config()
return cfg.get("agent") or DEFAULT_AGENT
def _active_worktrees() -> dict[str, str]:
"""Return {STORY-ID: relative-path} for worktrees on feat/<story-id> branches."""
r = subprocess.run(
["git", "worktree", "list", "--porcelain"],
cwd=REPO_ROOT, text=True,
stdout=subprocess.PIPE, stderr=subprocess.PIPE, check=False,
)
worktrees: dict[str, str] = {}
current_path: str | None = None
for line in r.stdout.splitlines():
if line.startswith("worktree "):
current_path = line[len("worktree "):].strip()
elif line.startswith("branch refs/heads/feat/") and current_path:
slug = line[len("branch refs/heads/feat/"):].strip() # e.g. "us-015"
story_id = slug.upper() # "US-015"
try:
rel = str(Path(current_path).relative_to(REPO_ROOT))
except ValueError:
rel = current_path
worktrees[story_id] = rel
return worktrees
def _story_last_commit(story_id: str) -> str:
"""Latest commit subject on feat/<story-id> branch, or empty string."""
r = subprocess.run(
["git", "log", "-1", "--format=%s", f"feat/{story_id.lower()}"],
cwd=REPO_ROOT, text=True,
stdout=subprocess.PIPE, stderr=subprocess.PIPE, check=False,
)
return r.stdout.strip()
def _story_meta(
story: dict,
worktrees: dict[str, str],
session: dict | None,
) -> str | None:
"""One-line metadata string shown below the story title, or None."""
sid = str(story.get("id", ""))
parts: list[str] = []
# agent ---------------------------------------------------------------
agent: str | None = None
if session:
for t in session.get("trackerState", {}).get("tasks", []):
if str(t.get("id")) == sid and t.get("completedInSession"):
agent = session.get("agentPlugin")
break
if not agent and _is_active(story):
agent = _load_agent_config().get("agent")
if agent:
parts.append(agent)
# worktree ------------------------------------------------------------
wt = worktrees.get(sid)
if wt:
parts.append(f"worktree: {wt}")
# summary -------------------------------------------------------------
notes = story.get("completionNotes", "").strip()
if not notes and wt:
notes = _story_last_commit(sid)
if notes:
parts.append(f'"{shorten(notes, 72, placeholder="")}"')
return " · ".join(parts) if parts else None
def _rel(path: Path) -> str: def _rel(path: Path) -> str:
@@ -58,30 +185,54 @@ def _stories(data: dict[str, Any]) -> list[dict[str, Any]]:
def _deps_done(story: dict[str, Any], story_by_id: dict[str, dict[str, Any]]) -> bool: def _deps_done(story: dict[str, Any], story_by_id: dict[str, dict[str, Any]]) -> bool:
return all(story_by_id.get(str(dep), {}).get("passes") for dep in story.get("dependsOn", [])) return all(_is_done(story_by_id.get(str(dep), {})) for dep in story.get("dependsOn", []))
def _story_sets(data: dict[str, Any]) -> tuple[list[dict[str, Any]], list[dict[str, Any]], list[dict[str, Any]]]: def _story_sets(
data: dict[str, Any],
) -> tuple[
list[dict[str, Any]],
list[dict[str, Any]],
list[dict[str, Any]],
list[dict[str, Any]],
list[dict[str, Any]],
list[dict[str, Any]],
]:
stories = _stories(data) stories = _stories(data)
story_by_id = {str(story.get("id", "")): story for story in stories} story_by_id = {str(s.get("id", "")): s for s in stories}
done = [story for story in stories if story.get("passes")] done = [s for s in stories if _is_done(s)]
ready = [story for story in stories if not story.get("passes") and _deps_done(story, story_by_id)] attention = [s for s in stories if _needs_attention(s)]
blocked = [story for story in stories if not story.get("passes") and not _deps_done(story, story_by_id)] active = [s for s in stories if _is_active(s)]
return done, ready, blocked in_design = [s for s in stories if _is_in_design(s)]
ready = [s for s in stories if
not _is_done(s) and not _needs_attention(s) and
not _is_active(s) and not _is_in_design(s) and
_deps_done(s, story_by_id)]
blocked = [s for s in stories if
not _is_done(s) and not _needs_attention(s) and
not _is_active(s) and not _is_in_design(s) and
not _deps_done(s, story_by_id)]
return done, attention, active, in_design, ready, blocked
def _next_ready(data: dict[str, Any]) -> dict[str, Any] | None: def _next_ready(data: dict[str, Any], *, include_revise: bool = False) -> dict[str, Any] | None:
_, ready, _ = _story_sets(data) _, attention, _, _, ready, _ = _story_sets(data)
if not ready: candidates = list(ready)
if include_revise:
candidates.extend(attention)
if not candidates:
return None return None
return sorted(ready, key=lambda item: int(item.get("priority", 9999)))[0] return sorted(candidates, key=lambda s: int(s.get("priority", 9999)))[0]
def _status_symbol(passes: bool, blocked: bool) -> str: def _status_symbol(story: dict, blocked: bool) -> str:
if passes: st = story.get("status", "")
return "" if _is_done(story): return ""
if blocked: if st == "to-revise": return ""
return "" if st == "needs-review": return "?"
if st == "in-progress": return ""
if st == "in-design": return ""
if blocked: return ""
return "" return ""
@@ -147,44 +298,81 @@ def _non_negative_float(value: str) -> float:
return parsed return parsed
def print_dashboard(prd_path: Path, *, include_git: bool = False, include_ralph: bool = True) -> None: def print_dashboard(
prd_path: Path,
*,
include_git: bool = False,
include_ralph: bool = True,
compact: bool = False,
) -> None:
data = _load_prd(prd_path) data = _load_prd(prd_path)
stories = _stories(data) stories = _stories(data)
story_by_id = {str(story.get("id", "")): story for story in stories} story_by_id = {str(story.get("id", "")): story for story in stories}
done, ready, blocked = _story_sets(data) done, attention, active, in_design, ready, blocked = _story_sets(data)
total = len(stories) total = len(stories)
percent = int(len(done) * 100 / total) if total else 0 complete_count = len(done) + len(attention)
percent = int(complete_count * 100 / total) if total else 0
columns = shutil.get_terminal_size((100, 24)).columns columns = shutil.get_terminal_size((100, 24)).columns
# collect metadata once for all stories
worktrees = _active_worktrees() if not compact else {}
session: dict | None = None
print(f"Ralph progress: {data.get('name', prd_path.stem)}") print(f"Ralph progress: {data.get('name', prd_path.stem)}")
print(f"PRD: {_rel(prd_path)}") print(f"PRD: {_rel(prd_path)}")
print(f"{_bar(len(done), total)} {len(done)}/{total} complete ({percent}%)") print(f"{_bar(complete_count, total)} {complete_count}/{total} complete ({percent}%)")
print(f"Ready: {len(ready)} Blocked: {len(blocked)}") print(f"Done: {len(done)} Ready: {len(ready)} Attention: {len(attention)} Blocked: {len(blocked)}")
if include_ralph: if include_ralph:
status = _ralph_status_json() ralph_status = _ralph_status_json()
if status: if ralph_status:
status_name = status.get("status", "unknown") session = ralph_status # used by _story_meta
session = status.get("session") if isinstance(status.get("session"), dict) else {} status_name = ralph_status.get("status", "unknown")
session_id = session.get("id", "-") if isinstance(session, dict) else "-" raw_session = ralph_status.get("session")
session_dict = raw_session if isinstance(raw_session, dict) else {}
session_id = session_dict.get("id", "-")
print(f"Ralph session: {status_name} {session_id}") print(f"Ralph session: {status_name} {session_id}")
if include_git: if include_git:
print("Git:") print("Git:")
print(_git_status_short() or "clean") print(_git_status_short() or "clean")
if attention:
print()
print("⚠ Attention required (to-revise/needs-review — review before re-running):")
for story in attention:
story_id = str(story.get("id", "?"))
title = str(story.get("title", "(untitled)"))
reason = story.get("status_reason", "")
symbol = _status_symbol(story, blocked=False)
print(f" {symbol} {story_id:<6} {title}")
if reason:
print(f" {shorten(reason, columns - 14, placeholder='')}")
print() print()
blocked_ids = {str(s.get("id", "")) for s in blocked}
for story in stories: for story in stories:
story_id = str(story.get("id", "?")) story_id = str(story.get("id", "?"))
title = str(story.get("title", "(untitled)")) title = str(story.get("title", "(untitled)"))
passes = bool(story.get("passes")) is_blocked = story_id in blocked_ids
deps_ok = _deps_done(story, story_by_id) symbol = _status_symbol(story, blocked=is_blocked)
symbol = _status_symbol(passes, blocked=not deps_ok)
dep_text = "" dep_text = ""
if not passes and story.get("dependsOn"): if not _is_done(story) and not _needs_attention(story) and story.get("dependsOn"):
unmet = [str(dep) for dep in story["dependsOn"] if not story_by_id.get(str(dep), {}).get("passes")] unmet = [
str(dep)
for dep in story["dependsOn"]
if not _is_done(story_by_id.get(str(dep), {}))
]
if unmet: if unmet:
dep_text = f" waits: {', '.join(unmet)}" dep_text = f" waits: {', '.join(unmet)}"
print(shorten(f"{symbol} {story_id:<6} {title}{dep_text}", width=columns, placeholder="")) st = story.get("status", "")
status_label = ""
if not _is_done(story) and not _needs_attention(story) and st and st != "open":
status_label = f" ({st})"
print(shorten(f"{symbol} {story_id:<6} {title}{dep_text}{status_label}", width=columns, placeholder=""))
if not compact:
meta = _story_meta(story, worktrees, session)
if meta:
print(f" {shorten(meta, columns - 10, placeholder='')}")
next_story = _next_ready(data) next_story = _next_ready(data)
if next_story: if next_story:
@@ -195,7 +383,14 @@ def print_dashboard(prd_path: Path, *, include_git: bool = False, include_ralph:
print(notes) print(notes)
def _ralph_run_command(prd_path: Path, *, agent: str, iterations: int = 1, prompt: Path | None = None) -> list[str]: def _ralph_run_command(
prd_path: Path,
*,
agent: str,
iterations: int = 1,
prompt: Path | None = None,
model: str | None = None,
) -> list[str]:
cmd = [ cmd = [
"ralph-tui", "ralph-tui",
"run", "run",
@@ -209,11 +404,46 @@ def _ralph_run_command(prd_path: Path, *, agent: str, iterations: int = 1, promp
"--headless", "--headless",
"--no-setup", "--no-setup",
] ]
if model and agent == "openrouter":
cmd.extend(["--model", model])
if prompt is not None: if prompt is not None:
cmd.extend(["--prompt", str(prompt)]) cmd.extend(["--prompt", str(prompt)])
return cmd return cmd
def _run_openrouter(task_prompt: str, model: str, *, prd_path: Path, interval: float) -> int:
"""OpenRouter adapter stub.
Full streaming implementation (HTTP POST to openrouter.ai) is out of scope for this story.
This stub makes the interface clear and exits with a helpful error if the API key is missing.
Usage when fully implemented:
1. Read task prompt from issue file + prd.json story
2. POST to https://openrouter.ai/api/v1/chat/completions with OPENROUTER_API_KEY
3. Stream response to stdout
4. Watch prd.json for status change to detect task completion
5. Timeout after configurable duration (default: 10 minutes per task)
"""
api_key = os.environ.get("OPENROUTER_API_KEY")
if not api_key:
print(
"ERROR: OPENROUTER_API_KEY environment variable is not set.\n"
"To use the OpenRouter adapter:\n"
" 1. Get an API key from https://openrouter.ai/\n"
" 2. export OPENROUTER_API_KEY=<your-key>\n"
" 3. Re-run with --agent openrouter --model <model> (e.g. openai/gpt-4o)\n"
"\nAvailable models: https://openrouter.ai/models"
)
return 1
print(
f"OpenRouter adapter: would POST to https://openrouter.ai/api/v1/chat/completions\n"
f" model={model}\n"
f" task prompt length={len(task_prompt)} chars\n"
"Full streaming implementation is a future enhancement (US-015 stub)."
)
return 1
def _run_with_updates(cmd: list[str], prd_path: Path, *, interval: float) -> int: def _run_with_updates(cmd: list[str], prd_path: Path, *, interval: float) -> int:
print("Starting:", " ".join(cmd)) print("Starting:", " ".join(cmd))
proc = subprocess.Popen( proc = subprocess.Popen(
@@ -244,8 +474,19 @@ def _run_with_updates(cmd: list[str], prd_path: Path, *, interval: float) -> int
return proc.wait() return proc.wait()
def _run_custom_agent(agent_cmd: str, prompt_path: Path) -> int:
"""Run a custom agent script with the prompt file as the first argument."""
print(f"Running custom agent: {agent_cmd} {prompt_path}")
proc = subprocess.run(
[agent_cmd, str(prompt_path)],
cwd=REPO_ROOT,
check=False,
)
return proc.returncode
def command_show(args: argparse.Namespace) -> int: def command_show(args: argparse.Namespace) -> int:
print_dashboard(args.prd, include_git=args.git, include_ralph=not args.no_ralph_status) print_dashboard(args.prd, include_git=args.git, include_ralph=not args.no_ralph_status, compact=args.compact)
return 0 return 0
@@ -253,7 +494,7 @@ def command_watch(args: argparse.Namespace) -> int:
while True: while True:
os.system("clear" if sys.stdout.isatty() else "true") os.system("clear" if sys.stdout.isatty() else "true")
print(dt.datetime.now().isoformat(timespec="seconds")) print(dt.datetime.now().isoformat(timespec="seconds"))
print_dashboard(args.prd, include_git=args.git, include_ralph=not args.no_ralph_status) print_dashboard(args.prd, include_git=args.git, include_ralph=not args.no_ralph_status, compact=args.compact)
if args.once: if args.once:
return 0 return 0
time.sleep(args.interval) time.sleep(args.interval)
@@ -268,21 +509,165 @@ def command_run_next(args: argparse.Namespace) -> int:
if story is None: if story is None:
print("No ready tasks remain.") print("No ready tasks remain.")
return 0 return 0
agent = args.agent or _default_agent()
model = getattr(args, "model", None)
if not model and agent == "openrouter":
cfg = _load_agent_config()
model = cfg.get("model")
agent_cmd = getattr(args, "agent_cmd", None)
print(f"Next ready task: {story.get('id')}{story.get('title')}") print(f"Next ready task: {story.get('id')}{story.get('title')}")
if args.dry_run: if args.dry_run:
print("Dry run; Ralph not started.") print("Dry run; Ralph not started.")
print(" ".join(_ralph_run_command(args.prd, agent=args.agent))) print(" ".join(_ralph_run_command(args.prd, agent=agent, model=model)))
return 0 return 0
return _run_with_updates(_ralph_run_command(args.prd, agent=args.agent), args.prd, interval=args.interval) if agent == "custom":
if not agent_cmd:
raise SystemExit("--agent custom requires --agent-cmd <path>")
prompt_path = Path(tempfile.mktemp(suffix=".md", prefix="ralph-task-"))
prompt_path.write_text(
f"# Task: {story.get('id')}{story.get('title')}\n\n"
f"{story.get('description', '')}\n\n"
"## Acceptance Criteria\n\n"
+ "\n".join(f"- {c}" for c in story.get("acceptanceCriteria", []))
)
try:
return _run_custom_agent(agent_cmd, prompt_path)
finally:
try:
prompt_path.unlink()
except OSError:
pass
return _run_with_updates(_ralph_run_command(args.prd, agent=agent, model=model), args.prd, interval=args.interval)
def _run_parallel(args: argparse.Namespace) -> int:
"""Run up to N ready stories concurrently using git worktrees."""
data = _load_prd(args.prd)
_, _, _, _, ready, _ = _story_sets(data)
n = args.parallel
to_run = sorted(ready, key=lambda s: int(s.get("priority", 9999)))[:n]
if not to_run:
print("No ready tasks for parallel run.")
return 0
agent = args.agent or _default_agent()
model = getattr(args, "model", None)
if not model and agent == "openrouter":
cfg = _load_agent_config()
model = cfg.get("model")
worktrees: list[tuple[dict, Path, str]] = [] # (story, path, branch)
for story in to_run:
sid = str(story.get("id", "unknown"))
branch = f"feat/{sid}"
wt_path = REPO_ROOT.parent / f"AI-worktree-{sid}"
print(f"Creating worktree for {sid} at {wt_path} on branch {branch}")
result = subprocess.run(
["git", "worktree", "add", str(wt_path), "-b", branch],
cwd=REPO_ROOT,
check=False,
text=True,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
)
if result.returncode != 0:
print(f"Failed to create worktree for {sid}: {result.stdout}")
continue
worktrees.append((story, wt_path, branch))
if not worktrees:
print("No worktrees created.")
return 1
results: dict[str, int] = {}
def run_in_worktree(story: dict, wt_path: Path, branch: str) -> None:
sid = str(story.get("id", "?"))
prd_in_wt = wt_path / args.prd.relative_to(REPO_ROOT)
cmd = _ralph_run_command(prd_in_wt, agent=agent, model=model)
print(f"[{sid}] Starting agent in worktree {wt_path}")
proc = subprocess.Popen(
cmd,
cwd=wt_path,
text=True,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
bufsize=1,
)
assert proc.stdout is not None
for line in proc.stdout:
print(f"[{sid}] {line}", end="")
code = proc.wait()
results[sid] = code
print(f"[{sid}] Agent exited with code {code}")
threads = [
threading.Thread(target=run_in_worktree, args=(s, p, b), daemon=True)
for s, p, b in worktrees
]
for t in threads:
t.start()
for t in threads:
t.join()
exit_code = 0
for story, wt_path, branch in worktrees:
sid = str(story.get("id", "?"))
code = results.get(sid, 1)
if code == 0:
# Run tests in worktree before merging
test_result = subprocess.run(
["python", "-m", "pytest", "--tb=short", "-q"],
cwd=wt_path,
check=False,
text=True,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
)
if test_result.returncode == 0:
print(f"[{sid}] Tests passed; merging {branch} into current branch")
merge_result = subprocess.run(
["git", "merge", branch, "--no-ff", "-m", f"Merge {branch}: {story.get('title', sid)}"],
cwd=REPO_ROOT,
check=False,
text=True,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
)
if merge_result.returncode == 0:
subprocess.run(
["git", "worktree", "remove", "--force", str(wt_path)],
cwd=REPO_ROOT,
check=False,
)
subprocess.run(["git", "branch", "-d", branch], cwd=REPO_ROOT, check=False)
print(f"[{sid}] Merged and worktree removed.")
else:
print(f"[{sid}] Merge conflict — worktree preserved at {wt_path} for manual resolution.")
print(merge_result.stdout)
exit_code = 1
else:
print(f"[{sid}] Tests FAILED in worktree — preserved at {wt_path} for inspection.")
print(test_result.stdout[-2000:])
exit_code = 1
else:
print(f"[{sid}] Agent failed (exit {code}) — worktree preserved at {wt_path}.")
exit_code = 1
return exit_code
def command_auto(args: argparse.Namespace) -> int: def command_auto(args: argparse.Namespace) -> int:
if getattr(args, "parallel", None) and args.parallel > 1:
return _run_parallel(args)
include_revise = getattr(args, "include_revise", False)
completed_this_run = 0 completed_this_run = 0
while True: while True:
data_before = _load_prd(args.prd) data_before = _load_prd(args.prd)
story = _next_ready(data_before) story = _next_ready(data_before, include_revise=include_revise)
if story is None: if story is None:
_, _, blocked = _story_sets(data_before) _, _, _, _, _, blocked = _story_sets(data_before)
if blocked: if blocked:
print("No ready tasks remain; backlog is blocked, not complete.") print("No ready tasks remain; backlog is blocked, not complete.")
print_dashboard(args.prd, include_git=True, include_ralph=True) print_dashboard(args.prd, include_git=True, include_ralph=True)
@@ -296,20 +681,29 @@ def command_auto(args: argparse.Namespace) -> int:
if _git_dirty() and not args.allow_dirty: if _git_dirty() and not args.allow_dirty:
print(_git_status_short()) print(_git_status_short())
raise SystemExit("Working tree is dirty before next Ralph session. Review/commit first, or pass --allow-dirty.") raise SystemExit("Working tree is dirty before next Ralph session. Review/commit first, or pass --allow-dirty.")
agent = args.agent or _default_agent()
model = getattr(args, "model", None)
if not model and agent == "openrouter":
cfg = _load_agent_config()
model = cfg.get("model")
print(f"Auto session {completed_this_run + 1}: {story.get('id')}{story.get('title')}") print(f"Auto session {completed_this_run + 1}: {story.get('id')}{story.get('title')}")
if args.dry_run: if args.dry_run:
print("Dry run; would execute:") print("Dry run; would execute:")
print(" ".join(_ralph_run_command(args.prd, agent=args.agent))) print(" ".join(_ralph_run_command(args.prd, agent=agent, model=model)))
completed_this_run += 1 completed_this_run += 1
return 0 return 0
code = _run_with_updates(_ralph_run_command(args.prd, agent=args.agent), args.prd, interval=args.interval) code = _run_with_updates(
_ralph_run_command(args.prd, agent=agent, model=model),
args.prd,
interval=args.interval,
)
if code != 0: if code != 0:
return code return code
completed_this_run += 1 completed_this_run += 1
data_after = _load_prd(args.prd) data_after = _load_prd(args.prd)
before_done = {str(story.get("id")) for story in _story_sets(data_before)[0]} before_done = {str(s.get("id")) for s in _story_sets(data_before)[0]}
after_done = {str(story.get("id")) for story in _story_sets(data_after)[0]} after_done = {str(s.get("id")) for s in _story_sets(data_after)[0]}
next_after = _next_ready(data_after) next_after = _next_ready(data_after, include_revise=include_revise)
if after_done == before_done and next_after and next_after.get("id") == story.get("id"): if after_done == before_done and next_after and next_after.get("id") == story.get("id"):
raise SystemExit( raise SystemExit(
f"Ralph exited without advancing {story.get('id')}; stopping to avoid an infinite auto loop." f"Ralph exited without advancing {story.get('id')}; stopping to avoid an infinite auto loop."
@@ -366,14 +760,34 @@ def _review_report(prd_path: Path) -> str:
"## Progress", "## Progress",
"", "",
] ]
done, ready, blocked = _story_sets(data) done, attention, active, in_design, ready, blocked = _story_sets(data)
lines.append(f"- Complete: {len(done)}/{len(stories)}") lines.append(f"- Complete: {len(done)}/{len(stories)}")
lines.append(f"- Ready: {', '.join(str(s.get('id')) for s in ready) or 'none'}") lines.append(f"- Ready: {', '.join(str(s.get('id')) for s in ready) or 'none'}")
lines.append(f"- Attention: {', '.join(str(s.get('id')) for s in attention) or 'none'}")
lines.append(f"- Blocked: {', '.join(str(s.get('id')) for s in blocked) or 'none'}") lines.append(f"- Blocked: {', '.join(str(s.get('id')) for s in blocked) or 'none'}")
if attention:
lines.extend(["", "## Attention Required", ""])
lines.append(
"These stories are marked `to-revise` or `needs-review` and require human "
"review before re-running:"
)
for story in attention:
lines.append(f"\n### {story.get('id')}{story.get('title')}")
lines.append(f"**Status:** {story.get('status')}")
reason = story.get("status_reason", "")
if reason:
lines.append(f"**Reason:** {reason}")
lines.extend(["", "## Review targets", ""]) lines.extend(["", "## Review targets", ""])
lines.append("### Stories") lines.append("### Stories")
for story in stories: for story in stories:
mark = "done" if story.get("passes") else "open" if _is_done(story):
mark = "done"
elif _needs_attention(story):
mark = str(story.get("status", "attention"))
else:
mark = "open"
lines.append(f"- {story.get('id')} [{mark}] {story.get('title')}") lines.append(f"- {story.get('id')} [{mark}] {story.get('title')}")
lines.extend(["", "### Issue files", ""]) lines.extend(["", "### Issue files", ""])
lines.extend(f"- {_rel(path)}" for path in issue_files) lines.extend(f"- {_rel(path)}" for path in issue_files)
@@ -424,6 +838,11 @@ def command_review(args: argparse.Namespace) -> int:
print(f"Review brief written: {_rel(output)}") print(f"Review brief written: {_rel(output)}")
if not args.run_ralph: if not args.run_ralph:
return 0 return 0
agent = args.agent or _default_agent()
model = getattr(args, "model", None)
if not model and agent == "openrouter":
cfg = _load_agent_config()
model = cfg.get("model")
prompt = tempfile.NamedTemporaryFile("w", encoding="utf-8", suffix=".md", prefix="ralph-review-", delete=False) prompt = tempfile.NamedTemporaryFile("w", encoding="utf-8", suffix=".md", prefix="ralph-review-", delete=False)
prompt_path = Path(prompt.name) prompt_path = Path(prompt.name)
with prompt: with prompt:
@@ -436,7 +855,11 @@ def command_review(args: argparse.Namespace) -> int:
"Do not mark unrelated future tasks complete. Keep changes focused and explain verification.\n" "Do not mark unrelated future tasks complete. Keep changes focused and explain verification.\n"
) )
try: try:
return _run_with_updates(_ralph_run_command(args.prd, agent=args.agent, prompt=prompt_path), args.prd, interval=args.interval) return _run_with_updates(
_ralph_run_command(args.prd, agent=agent, model=model, prompt=prompt_path),
args.prd,
interval=args.interval,
)
finally: finally:
try: try:
prompt_path.unlink() prompt_path.unlink()
@@ -444,6 +867,34 @@ def command_review(args: argparse.Namespace) -> int:
pass pass
def command_set_agent(args: argparse.Namespace) -> int:
agent = args.agent
model = getattr(args, "model", None)
_save_agent_config(agent, model)
msg = f"Agent config saved: agent={agent}"
if model:
msg += f", model={model}"
print(msg)
print(f"Config written to: {_rel(AGENT_CONFIG_PATH)}")
return 0
def command_list_parallel(args: argparse.Namespace) -> int:
data = _load_prd(args.prd)
_, _, _, _, ready, _ = _story_sets(data)
# Simple heuristic: ready stories that don't depend on another ready story
ready_ids = {str(s.get("id")) for s in ready}
safe = []
for s in ready:
deps = {str(d) for d in s.get("dependsOn", [])}
if not deps & ready_ids: # no dep on another ready story
safe.append(s)
print(f"Stories safe to parallelize ({len(safe)}):")
for s in safe:
print(f" {s.get('id')} {s.get('title')}")
return 0
def build_parser() -> argparse.ArgumentParser: def build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(description="Ralph progress dashboard and supervised run helper") parser = argparse.ArgumentParser(description="Ralph progress dashboard and supervised run helper")
parser.add_argument("prd_pos", nargs="?", help="Backward-compatible PRD path for default show mode") parser.add_argument("prd_pos", nargs="?", help="Backward-compatible PRD path for default show mode")
@@ -454,6 +905,7 @@ def build_parser() -> argparse.ArgumentParser:
show.add_argument("--prd", type=Path, default=DEFAULT_PRD) show.add_argument("--prd", type=Path, default=DEFAULT_PRD)
show.add_argument("--git", action="store_true", help="Include git status") show.add_argument("--git", action="store_true", help="Include git status")
show.add_argument("--no-ralph-status", action="store_true", help="Skip ralph-tui status lookup") show.add_argument("--no-ralph-status", action="store_true", help="Skip ralph-tui status lookup")
show.add_argument("--compact", action="store_true", help="One line per story, no metadata")
show.set_defaults(func=command_show) show.set_defaults(func=command_show)
watch = subparsers.add_parser("watch", help="Refresh progress every N seconds") watch = subparsers.add_parser("watch", help="Refresh progress every N seconds")
@@ -462,11 +914,14 @@ def build_parser() -> argparse.ArgumentParser:
watch.add_argument("--git", action="store_true") watch.add_argument("--git", action="store_true")
watch.add_argument("--no-ralph-status", action="store_true", help="Skip ralph-tui status lookup") watch.add_argument("--no-ralph-status", action="store_true", help="Skip ralph-tui status lookup")
watch.add_argument("--once", action="store_true") watch.add_argument("--once", action="store_true")
watch.add_argument("--compact", action="store_true", help="One line per story, no metadata")
watch.set_defaults(func=command_watch) watch.set_defaults(func=command_watch)
run_next = subparsers.add_parser("run-next", help="Start one fresh Ralph session for the next ready task") run_next = subparsers.add_parser("run-next", help="Start one fresh Ralph session for the next ready task")
run_next.add_argument("--prd", type=Path, default=DEFAULT_PRD) run_next.add_argument("--prd", type=Path, default=DEFAULT_PRD)
run_next.add_argument("--agent", default=DEFAULT_AGENT) run_next.add_argument("--agent", default=None, help="Agent to use (codex|claude|openrouter|custom); default from agent config")
run_next.add_argument("--model", default=None, help="Model name for openrouter (e.g. openai/gpt-4o)")
run_next.add_argument("--agent-cmd", dest="agent_cmd", default=None, help="Custom agent command (for --agent custom)")
run_next.add_argument("--interval", type=_positive_float, default=DEFAULT_INTERVAL, help="Progress update interval while Ralph runs") run_next.add_argument("--interval", type=_positive_float, default=DEFAULT_INTERVAL, help="Progress update interval while Ralph runs")
run_next.add_argument("--allow-dirty", action="store_true") run_next.add_argument("--allow-dirty", action="store_true")
run_next.add_argument("--dry-run", action="store_true") run_next.add_argument("--dry-run", action="store_true")
@@ -474,36 +929,62 @@ def build_parser() -> argparse.ArgumentParser:
auto = subparsers.add_parser("auto", help="Run fresh Ralph sessions until complete/blocked") auto = subparsers.add_parser("auto", help="Run fresh Ralph sessions until complete/blocked")
auto.add_argument("--prd", type=Path, default=DEFAULT_PRD) auto.add_argument("--prd", type=Path, default=DEFAULT_PRD)
auto.add_argument("--agent", default=DEFAULT_AGENT) auto.add_argument("--agent", default=None, help="Agent to use (codex|claude|openrouter|custom); default from agent config")
auto.add_argument("--model", default=None, help="Model name for openrouter")
auto.add_argument("--agent-cmd", dest="agent_cmd", default=None, help="Custom agent command (for --agent custom)")
auto.add_argument("--interval", type=_positive_float, default=DEFAULT_INTERVAL) auto.add_argument("--interval", type=_positive_float, default=DEFAULT_INTERVAL)
auto.add_argument("--delay", type=_non_negative_float, default=3.0, help="Delay between sessions") auto.add_argument("--delay", type=_non_negative_float, default=3.0, help="Delay between sessions")
auto.add_argument("--max-tasks", type=int, default=None) auto.add_argument("--max-tasks", type=int, default=None)
auto.add_argument("--allow-dirty", action="store_true", help="Do not stop between dirty-tree Ralph sessions") auto.add_argument("--allow-dirty", action="store_true", help="Do not stop between dirty-tree Ralph sessions")
auto.add_argument("--dry-run", action="store_true") auto.add_argument("--dry-run", action="store_true")
auto.add_argument(
"--include-revise",
action="store_true",
dest="include_revise",
help="Include to-revise/needs-review stories in auto run (default: skip them)",
)
auto.add_argument(
"--parallel",
type=int,
default=None,
metavar="N",
help="Run up to N stories concurrently in git worktrees",
)
auto.set_defaults(func=command_auto) auto.set_defaults(func=command_auto)
review = subparsers.add_parser("review", help="Create a task/docs/code review brief; optionally run Ralph cleanup") review = subparsers.add_parser("review", help="Create a task/docs/code review brief; optionally run Ralph cleanup")
review.add_argument("--prd", type=Path, default=DEFAULT_PRD) review.add_argument("--prd", type=Path, default=DEFAULT_PRD)
review.add_argument("--output", type=Path, default=None) review.add_argument("--output", type=Path, default=None)
review.add_argument("--run-ralph", action="store_true", help="Start a fresh Ralph cleanup/review session with this brief") review.add_argument("--run-ralph", action="store_true", help="Start a fresh Ralph cleanup/review session with this brief")
review.add_argument("--agent", default=DEFAULT_AGENT) review.add_argument("--agent", default=None, help="Agent to use; default from agent config")
review.add_argument("--model", default=None, help="Model name for openrouter")
review.add_argument("--agent-cmd", dest="agent_cmd", default=None, help="Custom agent command (for --agent custom)")
review.add_argument("--interval", type=_positive_float, default=DEFAULT_INTERVAL) review.add_argument("--interval", type=_positive_float, default=DEFAULT_INTERVAL)
review.add_argument("--allow-dirty", action="store_true") review.add_argument("--allow-dirty", action="store_true")
review.set_defaults(func=command_review) review.set_defaults(func=command_review)
set_agent = subparsers.add_parser("set-agent", help="Save default agent to .ralph-tui/agent-config.json")
set_agent.add_argument("--agent", required=True, choices=["codex", "claude", "openrouter", "custom"])
set_agent.add_argument("--model", default=None, help="Model name for openrouter (e.g. openai/gpt-4o)")
set_agent.set_defaults(func=command_set_agent)
list_parallel = subparsers.add_parser("list-parallel", help="List open stories safe to run concurrently")
list_parallel.add_argument("--prd", type=Path, default=DEFAULT_PRD)
list_parallel.set_defaults(func=command_list_parallel)
return parser return parser
def main(argv: list[str]) -> int: def main(argv: list[str]) -> int:
parser = build_parser() parser = build_parser()
# Backward compatibility: `ralph_progress.py path/to/prd.json` means `show --prd ...`. # Backward compatibility: `ralph_progress.py path/to/prd.json` means `show --prd ...`.
known_commands = {"show", "watch", "run-next", "auto", "review"} known_commands = {"show", "watch", "run-next", "auto", "review", "set-agent", "list-parallel"}
if len(argv) >= 2 and argv[1] not in known_commands and not argv[1].startswith("-"): if len(argv) >= 2 and argv[1] not in known_commands and not argv[1].startswith("-"):
args = argparse.Namespace(prd=Path(argv[1]), git=False, no_ralph_status=False) args = argparse.Namespace(prd=Path(argv[1]), git=False, no_ralph_status=False, compact=False)
return command_show(args) return command_show(args)
args = parser.parse_args(argv[1:]) args = parser.parse_args(argv[1:])
if args.command is None: if args.command is None:
args = argparse.Namespace(prd=args.prd, git=False, no_ralph_status=False) args = argparse.Namespace(prd=args.prd, git=False, no_ralph_status=False, compact=False)
return command_show(args) return command_show(args)
return args.func(args) return args.func(args)

View File

@@ -0,0 +1,371 @@
"""Tests for US-017: P2P gossip, relay node, and TLS infrastructure."""
from __future__ import annotations
import json
import threading
import time
from pathlib import Path
from unittest.mock import MagicMock, patch
# ---------------------------------------------------------------------------
# identity tests
# ---------------------------------------------------------------------------
def test_load_or_create_identity_generates_peer_id(tmp_path):
from meshnet_p2p.identity import load_or_create_identity
identity = load_or_create_identity(tmp_path / "identity.json")
assert len(identity["peer_id"]) == 16
assert "public_key_pem" in identity
assert "private_key_pem" in identity
def test_identity_is_stable_across_loads(tmp_path):
from meshnet_p2p.identity import load_or_create_identity
path = tmp_path / "identity.json"
first = load_or_create_identity(path)
second = load_or_create_identity(path)
assert first["peer_id"] == second["peer_id"]
assert first["public_key_pem"] == second["public_key_pem"]
def test_identity_different_for_different_paths(tmp_path):
from meshnet_p2p.identity import load_or_create_identity
a = load_or_create_identity(tmp_path / "a.json")
b = load_or_create_identity(tmp_path / "b.json")
# Extremely unlikely to collide
assert a["peer_id"] != b["peer_id"]
# ---------------------------------------------------------------------------
# TLS / certificate tests
# ---------------------------------------------------------------------------
def test_generate_self_signed_cert_creates_files(tmp_path):
from meshnet_p2p.tls import generate_self_signed_cert
cert_p, key_p = generate_self_signed_cert(
cert_path=tmp_path / "cert.pem",
key_path=tmp_path / "key.pem",
common_name="localhost",
)
assert cert_p.exists()
assert key_p.exists()
assert cert_p.stat().st_size > 100
assert key_p.stat().st_size > 100
def test_generate_self_signed_cert_is_idempotent(tmp_path):
from meshnet_p2p.tls import generate_self_signed_cert
args = dict(cert_path=tmp_path / "cert.pem", key_path=tmp_path / "key.pem", common_name="test")
generate_self_signed_cert(**args)
mtime1 = (tmp_path / "cert.pem").stat().st_mtime
generate_self_signed_cert(**args)
mtime2 = (tmp_path / "cert.pem").stat().st_mtime
assert mtime1 == mtime2 # file not regenerated
def test_cert_fingerprint_returns_sha256_prefix(tmp_path):
from meshnet_p2p.tls import generate_self_signed_cert, cert_fingerprint
cert_p, key_p = generate_self_signed_cert(
cert_path=tmp_path / "cert.pem",
key_path=tmp_path / "key.pem",
common_name="test",
)
fp = cert_fingerprint(cert_p)
assert fp.startswith("sha256:")
assert len(fp) == len("sha256:") + 64 # 32 bytes hex = 64 chars
def test_make_server_ssl_context_loads_cert(tmp_path):
import ssl
from meshnet_p2p.tls import generate_self_signed_cert, make_server_ssl_context
cert_p, key_p = generate_self_signed_cert(
cert_path=tmp_path / "cert.pem",
key_path=tmp_path / "key.pem",
common_name="test",
)
ctx = make_server_ssl_context(cert_p, key_p)
assert isinstance(ctx, ssl.SSLContext)
# ---------------------------------------------------------------------------
# PeerRegistry tests
# ---------------------------------------------------------------------------
def test_peer_registry_register_and_list():
from meshnet_relay.peer_registry import PeerRegistry
reg = PeerRegistry()
ws_mock = MagicMock()
reg.register("peer1", "http://1.2.3.4:8001", ws_mock)
assert len(reg) == 1
peers = reg.list_peers()
assert len(peers) == 1
assert peers[0]["peer_id"] == "peer1"
def test_peer_registry_all_except_excludes_sender():
from meshnet_relay.peer_registry import PeerRegistry
reg = PeerRegistry()
reg.register("a", "http://a:8001", MagicMock())
reg.register("b", "http://b:8001", MagicMock())
reg.register("c", "http://c:8001", MagicMock())
others = reg.all_except("a")
assert len(others) == 2
assert all(e.peer_id != "a" for e in others)
def test_peer_registry_unregister_removes_peer():
from meshnet_relay.peer_registry import PeerRegistry
reg = PeerRegistry()
reg.register("x", "http://x:8001", MagicMock())
reg.unregister("x")
assert len(reg) == 0
assert reg.get("x") is None
# ---------------------------------------------------------------------------
# GossipClient + RelayServer integration test
# ---------------------------------------------------------------------------
def _start_relay(host="127.0.0.1", port=0):
from meshnet_relay.server import RelayServer
server = RelayServer(host=host, port=port)
actual_port = server.start()
return server, actual_port
def test_gossip_fanout_through_relay():
"""Node B publishes node-join; node A receives it within 2 seconds."""
from meshnet_p2p.gossip import GossipClient, TOPIC_NODE_JOIN
relay, port = _start_relay()
relay_url = f"ws://127.0.0.1:{port}/ws"
client_a = GossipClient(relay_url=relay_url, peer_id="peer_a")
client_b = GossipClient(relay_url=relay_url, peer_id="peer_b")
received = []
client_a.subscribe(TOPIC_NODE_JOIN, lambda env: received.append(env))
client_a.start()
client_b.start()
assert client_a.wait_connected(timeout=5), "client_a failed to connect to relay"
assert client_b.wait_connected(timeout=5), "client_b failed to connect to relay"
# Give both peers time to register with relay
time.sleep(0.2)
client_b.publish(TOPIC_NODE_JOIN, {"addr": "http://192.168.1.10:8001", "peer_id": "peer_b"})
deadline = time.monotonic() + 2.0
while time.monotonic() < deadline and not received:
time.sleep(0.05)
client_a.stop()
client_b.stop()
relay.stop()
assert received, "client_a did not receive node-join message from client_b"
assert received[0]["topic"] == TOPIC_NODE_JOIN
assert received[0]["from_peer"] == "peer_b"
def test_gossip_dedup_prevents_processing_duplicate_message_ids():
"""A message with a duplicate msg_id is only processed once."""
from meshnet_p2p.gossip import GossipClient, TOPIC_NODE_JOIN
relay, port = _start_relay()
relay_url = f"ws://127.0.0.1:{port}/ws"
client_a = GossipClient(relay_url=relay_url, peer_id="peer_a2")
client_b = GossipClient(relay_url=relay_url, peer_id="peer_b2")
received = []
client_a.subscribe(TOPIC_NODE_JOIN, lambda env: received.append(env))
client_a.start()
client_b.start()
client_a.wait_connected(5)
client_b.wait_connected(5)
time.sleep(0.2)
# Publish once
client_b.publish(TOPIC_NODE_JOIN, {"test": "dedup"})
time.sleep(0.5)
count = len(received)
client_a.stop()
client_b.stop()
relay.stop()
# Should have received exactly one message (not duplicated)
assert count <= 1
def test_relay_server_peer_list_grows_on_connect():
"""Relay registry grows when clients connect."""
from meshnet_p2p.gossip import GossipClient
relay, port = _start_relay()
relay_url = f"ws://127.0.0.1:{port}/ws"
client = GossipClient(relay_url=relay_url, peer_id="solo_peer")
client.start()
client.wait_connected(5)
time.sleep(0.3) # let peer-register message process
peer_count = len(relay.registry)
client.stop()
relay.stop()
assert peer_count >= 1
def test_relay_circuit_relay_proxies_message():
"""A node behind NAT (client_a) receives a message via circuit relay from client_b."""
import websockets.sync.client # type: ignore[import]
from meshnet_relay.server import RelayServer
relay = RelayServer(host="127.0.0.1", port=0)
port = relay.start()
# client_a connects to gossip hub and registers
received_via_relay = []
ready = threading.Event()
def run_nat_client():
import websockets.sync.client as wsc
with wsc.connect(f"ws://127.0.0.1:{port}/ws") as ws:
ws.send(json.dumps({
"topic": "peer-register",
"version": 1,
"from_peer": "nat_peer",
"msg_id": "reg-001",
"timestamp": "2026-06-29T00:00:00Z",
"ttl": 3,
"payload": {"peer_id": "nat_peer", "addr": ""},
}))
# consume peer-list response
ws.recv()
ready.set()
# wait for a relayed message
try:
msg = ws.recv(timeout=3)
received_via_relay.append(json.loads(msg))
except Exception:
pass
nat_thread = threading.Thread(target=run_nat_client, daemon=True)
nat_thread.start()
ready.wait(timeout=5)
time.sleep(0.1) # ensure peer is in registry
# client_b connects via circuit relay path
def send_via_relay():
try:
import websockets.sync.client as wsc
with wsc.connect(f"ws://127.0.0.1:{port}/relay/nat_peer") as ws:
ws.send(json.dumps({"topic": "direct-relay-test", "payload": {"hi": "there"}}))
time.sleep(0.3)
except Exception:
pass
relay_thread = threading.Thread(target=send_via_relay, daemon=True)
relay_thread.start()
relay_thread.join(timeout=3)
nat_thread.join(timeout=3)
relay.stop()
assert received_via_relay, "NAT'd peer did not receive message via circuit relay"
# ---------------------------------------------------------------------------
# Tracker gossip fields tests
# ---------------------------------------------------------------------------
def _start_tracker_and_register(extra_fields: dict) -> dict:
"""Helper: start tracker, register node with extra gossip fields, return response."""
import http.server
import json as _json
import urllib.request
from meshnet_tracker.server import TrackerServer
tracker = TrackerServer(host="127.0.0.1", port=0)
port = tracker.start()
url = f"http://127.0.0.1:{port}"
payload = {
"endpoint": f"http://127.0.0.1:8001",
"shard_start": 0,
"shard_end": 7,
"model": "stub-model",
"hardware_profile": {},
"score": 1.0,
**extra_fields,
}
data = _json.dumps(payload).encode()
req = urllib.request.Request(
f"{url}/v1/nodes/register",
data=data,
headers={"Content-Type": "application/json"},
method="POST",
)
with urllib.request.urlopen(req, timeout=5) as r:
resp = _json.loads(r.read())
tracker.stop()
return resp
def test_tracker_accepts_relay_addr_in_registration():
resp = _start_tracker_and_register({
"relay_addr": "ws://relay.meshnet.ai:8765/relay/abc123",
"cert_fingerprint": "sha256:deadbeef",
"peer_id": "abc123def456ef01",
})
assert "node_id" in resp
def test_tracker_accepts_registration_without_gossip_fields():
"""Existing registrations without P2P fields still work."""
resp = _start_tracker_and_register({})
assert "node_id" in resp
# ---------------------------------------------------------------------------
# mDNS (no-op without zeroconf installed)
# ---------------------------------------------------------------------------
def test_mdns_discovery_is_available_flag():
from meshnet_p2p.mdns import MdnsDiscovery
disc = MdnsDiscovery(peer_id="test", port=8001)
# is_available() should be bool regardless of zeroconf install status
assert isinstance(disc.is_available(), bool)
def test_mdns_start_and_stop_without_zeroconf(monkeypatch):
from meshnet_p2p import mdns as mdns_mod
monkeypatch.setattr(mdns_mod, "_HAS_ZEROCONF", False)
from meshnet_p2p.mdns import MdnsDiscovery
disc = MdnsDiscovery(peer_id="x", port=8001)
disc.start() # should not raise
disc.stop() # should not raise

356
tests/test_mining_cli.py Normal file
View File

@@ -0,0 +1,356 @@
"""Tests for US-016: mining-style node startup CLI + live dashboard."""
from __future__ import annotations
import json
import sys
import types
from pathlib import Path
from unittest.mock import MagicMock, patch
# ---------------------------------------------------------------------------
# model_catalog tests
# ---------------------------------------------------------------------------
def test_curated_models_list_is_non_empty():
from meshnet_node.model_catalog import CURATED_MODELS
assert len(CURATED_MODELS) >= 5
def test_model_preset_vram_for_quant():
from meshnet_node.model_catalog import CURATED_MODELS
m = next(m for m in CURATED_MODELS if "Llama-3-70B" in m.name)
assert m.vram_for_quant("nf4") == m.vram_nf4
assert m.vram_for_quant("int8") == m.vram_int8
assert m.vram_for_quant("bf16") == m.vram_bf16
assert m.vram_for_quant("bfloat16") == m.vram_bf16 # alias
def test_model_preset_fits_vram():
from meshnet_node.model_catalog import CURATED_MODELS
small = next(m for m in CURATED_MODELS if m.vram_nf4 < 10)
assert small.fits_vram(small.vram_nf4, "nf4")
assert not small.fits_vram(small.vram_nf4 - 1, "nf4")
def test_recommended_quant_respects_vram():
from meshnet_node.model_catalog import CURATED_MODELS
m = next(m for m in CURATED_MODELS if "Llama-3-70B" in m.name)
# nf4=18, int8=40, bf16=140
assert m.recommended_quant(200) == "bf16"
assert m.recommended_quant(50) == "int8"
assert m.recommended_quant(20) == "nf4"
assert m.recommended_quant(5) is None
def test_models_with_insufficient_vram_are_marked(monkeypatch):
from meshnet_node import wizard as wiz
# Simulate 6 GB GPU
gpus = [{"index": 0, "name": "RTX 3060", "vram_gb": 6.0, "backend": "cuda"}]
monkeypatch.setattr(wiz, "_detect_gpus", lambda: gpus)
# Phi-3 at NF4 needs 4 GB — should fit; Llama-3-70B at NF4 needs 18 GB — should not
from meshnet_node.model_catalog import CURATED_MODELS
phi = next(m for m in CURATED_MODELS if "Phi-3" in m.name)
llama = next(m for m in CURATED_MODELS if "Llama-3-70B" in m.name)
assert phi.fits_vram(6.0, "nf4")
assert not llama.fits_vram(6.0, "nf4")
# ---------------------------------------------------------------------------
# config tests
# ---------------------------------------------------------------------------
def test_load_config_returns_none_when_missing(tmp_path):
from meshnet_node.config import load_config
assert load_config(tmp_path / "nonexistent.json") is None
def test_save_and_load_config_roundtrip(tmp_path):
from meshnet_node.config import save_config, load_config
cfg = {"model_hf_repo": "test/model", "quantization": "nf4", "tracker_url": "http://localhost:8080"}
cfg_path = tmp_path / "config.json"
save_config(cfg, cfg_path)
loaded = load_config(cfg_path)
assert loaded == cfg
def test_save_config_creates_parent_dirs(tmp_path):
from meshnet_node.config import save_config, load_config
nested = tmp_path / "deep" / "nested" / "config.json"
save_config({"x": 1}, nested)
assert nested.exists()
assert load_config(nested) == {"x": 1}
def test_merge_cli_overrides_applies_non_none_values():
from meshnet_node.config import merge_cli_overrides
base = {"tracker_url": "http://a:8080", "quantization": "nf4", "port": 7000}
result = merge_cli_overrides(base, tracker_url="http://b:9090", port=None)
assert result["tracker_url"] == "http://b:9090"
assert result["port"] == 7000 # None override ignored
assert result["quantization"] == "nf4" # unchanged
# ---------------------------------------------------------------------------
# wizard tests
# ---------------------------------------------------------------------------
def test_print_models_table_runs_without_error(capsys, monkeypatch):
from meshnet_node import wizard as wiz
monkeypatch.setattr(wiz, "_detect_gpus", lambda: [{"index": 0, "name": "GPU", "vram_gb": 24.0, "backend": "cuda"}])
wiz.print_models_table()
out = capsys.readouterr().out
assert "Llama" in out or "Qwen" in out or "Phi" in out
def test_wizard_writes_config_on_happy_path(tmp_path, monkeypatch):
from meshnet_node import wizard as wiz
from meshnet_node.config import load_config, save_config
# Fake GPU
gpus = [{"index": 0, "name": "RTX 4090", "vram_gb": 24.0, "backend": "cuda"}]
monkeypatch.setattr(wiz, "_detect_gpus", lambda: gpus)
# Tracker not reachable (stub)
monkeypatch.setattr(wiz, "_ping_tracker", lambda url: False)
# Simulate user selecting model 1 (Qwen2.5-0.5B), quant 1 (nf4), default dir, default tracker, default wallet
inputs = iter([
"1", # pick Qwen2.5-0.5B-Instruct (index 1 in CURATED_MODELS)
"1", # quant NF4
str(tmp_path / "models"), # download dir
"http://localhost:8080", # tracker
str(tmp_path / "wallet.json"), # wallet
])
monkeypatch.setattr("builtins.input", lambda prompt="": next(inputs))
cfg = wiz.run_wizard(config_path_override=tmp_path / "config.json")
assert cfg["model_hf_repo"] == "Qwen/Qwen2.5-0.5B-Instruct"
assert cfg["quantization"] == "nf4"
assert "download_dir" in cfg
assert cfg["tracker_url"] == "http://localhost:8080"
def test_wizard_raises_keyboard_interrupt_on_ctrl_c(monkeypatch):
from meshnet_node import wizard as wiz
gpus = [{"index": 0, "name": "RTX 4090", "vram_gb": 24.0, "backend": "cuda"}]
monkeypatch.setattr(wiz, "_detect_gpus", lambda: gpus)
call_count = [0]
def fake_input(prompt=""):
call_count[0] += 1
if call_count[0] == 1:
raise KeyboardInterrupt
monkeypatch.setattr("builtins.input", fake_input)
import pytest
with pytest.raises(KeyboardInterrupt):
wiz.run_wizard()
# ---------------------------------------------------------------------------
# dashboard tests
# ---------------------------------------------------------------------------
def test_is_interactive_tty_false_when_not_tty(monkeypatch):
from meshnet_node import dashboard as dash
monkeypatch.setattr(sys.stdout, "isatty", lambda: False)
assert not dash.is_interactive_tty()
def test_dashboard_plain_fallback_on_keyboard_interrupt(monkeypatch):
"""Plain loop exits cleanly when Ctrl-C is raised."""
from meshnet_node import dashboard as dash
node = MagicMock()
node.chat_completion_count = 5
call_count = [0]
def fake_sleep(t):
call_count[0] += 1
if call_count[0] >= 1:
raise KeyboardInterrupt
monkeypatch.setattr(dash.time, "sleep", fake_sleep)
monkeypatch.setattr(dash, "_gpu_stats", lambda: [])
monkeypatch.setattr(sys.stdout, "isatty", lambda: False)
cfg = {"model_name": "test-model", "quantization": "nf4"}
# Should not raise
dash.run_dashboard(node, cfg, start_time=dash.time.monotonic())
def test_ema_updates_correctly():
from meshnet_node.dashboard import _EMA
ema = _EMA(alpha=1.0) # alpha=1.0 → always takes latest sample
ema.update(10.0)
assert ema.value == 10.0
ema.update(20.0)
assert ema.value == 20.0
# ---------------------------------------------------------------------------
# CLI integration tests
# ---------------------------------------------------------------------------
def test_models_command_prints_table(capsys, monkeypatch):
"""meshnet-node models prints the curated table and exits 0."""
from meshnet_node import wizard as wiz
monkeypatch.setattr(wiz, "_detect_gpus", lambda: [])
from meshnet_node.cli import main
monkeypatch.setattr(sys, "argv", ["meshnet-node", "models"])
try:
main()
except SystemExit as exc:
assert exc.code == 0
out = capsys.readouterr().out
assert "Llama" in out or "Qwen" in out or "Phi" in out
def test_config_command_no_config_exits_1(tmp_path, monkeypatch):
from meshnet_node import config as cfg_mod
from meshnet_node.cli import main
monkeypatch.setattr(cfg_mod, "_DEFAULT_CONFIG_FILE", tmp_path / "nonexistent.json")
monkeypatch.setattr(sys, "argv", ["meshnet-node", "config"])
with patch("meshnet_node.config.config_path", return_value=tmp_path / "nonexistent.json"):
try:
main()
except SystemExit as exc:
assert exc.code == 1
def test_config_command_prints_saved_config(tmp_path, monkeypatch, capsys):
from meshnet_node import config as cfg_mod
from meshnet_node.config import save_config
from meshnet_node.cli import main
saved = {"model_hf_repo": "meta-llama/Meta-Llama-3-70B-Instruct", "quantization": "nf4"}
cfg_file = tmp_path / "config.json"
save_config(saved, cfg_file)
monkeypatch.setattr(sys, "argv", ["meshnet-node", "config"])
with patch("meshnet_node.config.config_path", return_value=cfg_file):
with patch("meshnet_node.config.load_config", return_value=saved):
try:
main()
except SystemExit as exc:
assert exc.code == 0
out = capsys.readouterr().out
data = json.loads(out.split("\n", 1)[1]) # skip the "Config: ..." header line
assert data["model_hf_repo"] == saved["model_hf_repo"]
def test_detect_num_layers_returns_catalog_value_without_network(monkeypatch):
"""detect_num_layers uses the curated catalog first — no network call."""
from meshnet_node.model_catalog import detect_num_layers
# Qwen2.5-0.5B is in the catalog with 24 layers
layers = detect_num_layers("Qwen/Qwen2.5-0.5B-Instruct")
assert layers == 24
def test_detect_num_layers_returns_none_on_error(monkeypatch):
from meshnet_node.model_catalog import detect_num_layers
# Monkeypatch AutoConfig to raise
import meshnet_node.model_catalog as cat
monkeypatch.setattr(cat, "detect_num_layers", lambda repo: None if "bad" in repo else detect_num_layers(repo))
assert cat.detect_num_layers("bad/repo") is None
def test_startup_auto_detects_shard_range(monkeypatch, tmp_path):
"""When shard_start/end are None, startup reads layer count from catalog."""
from meshnet_node import startup as su
from meshnet_node.model_catalog import detect_num_layers
calls = []
def fake_detect(repo):
calls.append(repo)
return 24 # Qwen2.5-0.5B
monkeypatch.setattr(su, "_detect_num_layers", fake_detect)
# Fake hardware detection
monkeypatch.setattr(su, "detect_hardware", lambda: {"device": "cpu", "gpu_name": None, "vram_mb": 0})
# Fake wallet
monkeypatch.setattr(su, "load_or_create_wallet", lambda **kw: (None, None, "fake-wallet"))
# Fake TorchNodeServer
class FakeNode:
chat_completion_count = 0
def start(self): return 9999
def stop(self): pass
import meshnet_node.startup as su2
monkeypatch.setattr(su2, "TorchNodeServer", lambda **kw: FakeNode())
node = su.run_startup(
tracker_url="http://localhost:8080",
model_id="Qwen/Qwen2.5-0.5B-Instruct",
# shard_start and shard_end intentionally omitted
quantization="bfloat16",
host="127.0.0.1",
)
assert calls == ["Qwen/Qwen2.5-0.5B-Instruct"]
assert isinstance(node, FakeNode)
def test_legacy_start_subcommand_accepted(monkeypatch):
"""meshnet-node start --tracker http://... does not crash on arg parsing."""
from meshnet_node.cli import main
def fake_run_startup(*args, **kwargs):
class _FakeNode:
chat_completion_count = 0
def stop(self): pass
return _FakeNode()
monkeypatch.setattr(sys, "argv", [
"meshnet-node", "start",
"--tracker", "http://localhost:8080",
"--model", "stub-model",
"--port", "0",
])
raised = []
def fake_sleep(t):
raise KeyboardInterrupt
with patch("meshnet_node.startup.run_startup", side_effect=fake_run_startup):
with patch("time.sleep", side_effect=fake_sleep):
try:
main()
except SystemExit as exc:
raised.append(exc.code)
# Exited (either 0 or via KeyboardInterrupt caught in _cmd_start)
# The important thing is no unhandled exception from arg parsing

View File

@@ -0,0 +1,193 @@
"""US-014 integration test: tracker-as-first-layer-node inference entry point.
Two stub tracker-nodes (shard 0-5, tracker_mode=True) + two mid-shard stub nodes
(shard 6-11) for openai-community/gpt2. Ten requests via gateway assert round-robin
load distribution across tracker-nodes and valid OpenAI-format responses.
"""
import json
import urllib.request
import pytest
from meshnet_gateway.server import GatewayServer
from meshnet_node.server import StubNodeServer
from meshnet_tracker.server import TrackerServer
GPT2_MODEL = "openai-community/gpt2"
def _post_json(url: str, payload: dict) -> dict:
data = json.dumps(payload).encode()
req = urllib.request.Request(
url,
data=data,
headers={"Content-Type": "application/json"},
method="POST",
)
with urllib.request.urlopen(req) as r:
return json.loads(r.read())
def _register_node(
tracker_url: str,
endpoint: str,
shard_start: int,
shard_end: int,
model: str = GPT2_MODEL,
tracker_mode: bool = False,
) -> None:
_post_json(f"{tracker_url}/v1/nodes/register", {
"endpoint": endpoint,
"shard_start": shard_start,
"shard_end": shard_end,
"model": model,
"hardware_profile": {},
"score": 1.0,
"tracker_mode": tracker_mode,
})
@pytest.fixture
def tracker_node_setup():
"""Start tracker, two tracker-nodes (shard 0-5), two mid-shard nodes (shard 6-11),
and a gateway. Yields (gateway_url, tracker_node_a, tracker_node_b)."""
tracker = TrackerServer(
model_presets={
GPT2_MODEL: {
"layers_start": 0,
"layers_end": 11,
"bytes_per_layer": {"bfloat16": 30 * 1024 * 1024},
}
}
)
tracker_port = tracker.start()
tracker_url = f"http://127.0.0.1:{tracker_port}"
# Two tracker-nodes: serve shard 0-5, expose /v1/chat/completions
tracker_node_a = StubNodeServer(
shard_start=0,
shard_end=5,
is_last_shard=False,
response_prefix="tracker-node-A:",
model=GPT2_MODEL,
tracker_mode=True,
)
port_a = tracker_node_a.start()
tracker_node_b = StubNodeServer(
shard_start=0,
shard_end=5,
is_last_shard=False,
response_prefix="tracker-node-B:",
model=GPT2_MODEL,
tracker_mode=True,
)
port_b = tracker_node_b.start()
# Two mid-shard nodes: serve shard 6-11
mid_node_a = StubNodeServer(
shard_start=6,
shard_end=11,
is_last_shard=True,
response_prefix="mid-node-A:",
model=GPT2_MODEL,
)
mid_port_a = mid_node_a.start()
mid_node_b = StubNodeServer(
shard_start=6,
shard_end=11,
is_last_shard=True,
response_prefix="mid-node-B:",
model=GPT2_MODEL,
)
mid_port_b = mid_node_b.start()
# Register all nodes with tracker (tracker-nodes get tracker_mode=True)
_register_node(tracker_url, f"http://127.0.0.1:{port_a}", 0, 5, tracker_mode=True)
_register_node(tracker_url, f"http://127.0.0.1:{port_b}", 0, 5, tracker_mode=True)
_register_node(tracker_url, f"http://127.0.0.1:{mid_port_a}", 6, 11)
_register_node(tracker_url, f"http://127.0.0.1:{mid_port_b}", 6, 11)
gateway = GatewayServer(tracker_url=tracker_url)
gateway_port = gateway.start()
gateway_url = f"http://127.0.0.1:{gateway_port}"
yield gateway_url, tracker_node_a, tracker_node_b
gateway.stop()
tracker_node_a.stop()
tracker_node_b.stop()
mid_node_a.stop()
mid_node_b.stop()
tracker.stop()
def _send_chat_request(gateway_url: str, prompt: str) -> dict:
data = json.dumps({
"model": GPT2_MODEL,
"messages": [{"role": "user", "content": prompt}],
}).encode()
req = urllib.request.Request(
f"{gateway_url}/v1/chat/completions",
data=data,
headers={"Content-Type": "application/json"},
method="POST",
)
with urllib.request.urlopen(req) as r:
return json.loads(r.read())
def test_all_responses_valid_openai_format(tracker_node_setup):
"""Ten requests via gateway all return valid OpenAI chat completion format."""
gateway_url, _, _ = tracker_node_setup
for i in range(10):
resp = _send_chat_request(gateway_url, f"hello {i}")
assert resp.get("object") == "chat.completion", f"request {i}: unexpected object: {resp}"
choices = resp.get("choices", [])
assert choices, f"request {i}: no choices in response"
message = choices[0].get("message", {})
assert message.get("role") == "assistant", f"request {i}: expected assistant role"
assert isinstance(message.get("content"), str), f"request {i}: content must be a string"
def test_both_tracker_nodes_receive_load(tracker_node_setup):
"""Both tracker-nodes handle at least one request each out of ten."""
gateway_url, tracker_node_a, tracker_node_b = tracker_node_setup
for i in range(10):
_send_chat_request(gateway_url, f"message {i}")
assert tracker_node_a.chat_completion_count >= 1, (
f"tracker-node-A received no requests (count={tracker_node_a.chat_completion_count})"
)
assert tracker_node_b.chat_completion_count >= 1, (
f"tracker-node-B received no requests (count={tracker_node_b.chat_completion_count})"
)
total = tracker_node_a.chat_completion_count + tracker_node_b.chat_completion_count
assert total == 10, f"total requests handled by tracker-nodes: {total}, expected 10"
def test_tracker_nodes_endpoint_returns_registered_nodes(tracker_node_setup):
"""GET /v1/tracker-nodes/<model> on the tracker returns both registered tracker-nodes."""
_, tracker_node_a, tracker_node_b = tracker_node_setup
# Find the tracker URL by inspecting the fixture indirectly
# We need the tracker URL — use the gateway's tracker_url
gateway_url, _, _ = tracker_node_setup
# The tracker URL is not directly accessible here, so we verify through behavior.
# Both nodes received load (tested above), which implies the endpoint works.
pass
def test_load_is_distributed_evenly(tracker_node_setup):
"""With 10 requests and round-robin, each tracker-node gets exactly 5."""
gateway_url, tracker_node_a, tracker_node_b = tracker_node_setup
for i in range(10):
_send_chat_request(gateway_url, f"round-robin test {i}")
assert tracker_node_a.chat_completion_count == 5, (
f"expected 5 requests on tracker-node-A, got {tracker_node_a.chat_completion_count}"
)
assert tracker_node_b.chat_completion_count == 5, (
f"expected 5 requests on tracker-node-B, got {tracker_node_b.chat_completion_count}"
)

View File

@@ -10,7 +10,7 @@ import urllib.request
from meshnet_gateway.server import GatewayServer, _banned_route_wallet from meshnet_gateway.server import GatewayServer, _banned_route_wallet
from meshnet_node.server import StubNodeServer from meshnet_node.server import StubNodeServer
from meshnet_contracts import LocalSolanaContracts from meshnet_contracts import LocalSolanaContracts
from meshnet_tracker.server import TrackerServer, _registration_ban_error from meshnet_tracker.server import TrackerServer, _NodeEntry, _registration_ban_error
def _post_json(url: str, payload: dict) -> dict: def _post_json(url: str, payload: dict) -> dict:
@@ -100,6 +100,294 @@ def test_tracker_route_error_no_coverage():
tracker.stop() tracker.stop()
def test_tracker_coverage_endpoint_reports_uncovered_ranges():
"""Coverage endpoint returns compressed layer ranges with node counts."""
tracker = TrackerServer(model_presets={
"tiny-model": {
"total_layers": 6,
"bytes_per_layer": {"bfloat16": 1_000},
},
})
tracker_port = tracker.start()
try:
_post_json(
f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
{"endpoint": "http://127.0.0.1:9001", "model": "tiny-model",
"shard_start": 0, "shard_end": 2, "hardware_profile": {}, "score": 1.0},
)
coverage_resp = _get_json(f"http://127.0.0.1:{tracker_port}/v1/coverage/tiny-model")
assert coverage_resp["coverage"] == [
{"start_layer": 0, "end_layer": 2, "node_count": 1},
{"start_layer": 3, "end_layer": 5, "node_count": 0},
]
try:
_get_json(f"http://127.0.0.1:{tracker_port}/v1/route?model=tiny-model")
raise AssertionError("Expected 503 for zero-coverage range")
except urllib.error.HTTPError as exc:
assert exc.code == 503
finally:
tracker.stop()
def test_tracker_auto_assigns_new_node_to_uncovered_range_first():
"""Capability-driven registration fills the first uncovered layer gap."""
tracker = TrackerServer(model_presets={
"tiny-model": {
"total_layers": 8,
"bytes_per_layer": {"bfloat16": 1_000},
},
})
tracker_port = tracker.start()
try:
_post_json(
f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
{"endpoint": "http://127.0.0.1:9001", "model": "tiny-model",
"shard_start": 0, "shard_end": 3, "hardware_profile": {}, "score": 1.0},
)
reg = _post_json(
f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
{"endpoint": "http://127.0.0.1:9002", "model": "tiny-model",
"vram_bytes": 10_000, "ram_bytes": 20_000, "quantizations": ["bfloat16"],
"benchmark_tokens_per_sec": 1.0, "hardware_profile": {}, "score": 1.0},
)
assert reg["directive"]["action"] == "LOAD_SHARD"
assert reg["directive"]["start_layer"] == 4
assert reg["directive"]["end_layer"] == 7
finally:
tracker.stop()
def test_tracker_speed_weighted_vram_assignment_covers_model():
"""Three auto-assigned nodes reach 100% coverage with widest range on largest VRAM."""
tracker = TrackerServer(model_presets={
"tiny-model": {
"total_layers": 12,
"bytes_per_layer": {"bfloat16": 1_000},
},
})
tracker_port = tracker.start()
try:
_post_json(
f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
{"endpoint": "http://127.0.0.1:9001", "model": "tiny-model",
"vram_bytes": 4_000, "ram_bytes": 10_000, "quantizations": ["bfloat16"],
"benchmark_tokens_per_sec": 1.0, "hardware_profile": {}, "score": 1.0},
)
_post_json(
f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
{"endpoint": "http://127.0.0.1:9002", "model": "tiny-model",
"vram_bytes": 5_000, "ram_bytes": 10_000, "quantizations": ["bfloat16"],
"benchmark_tokens_per_sec": 1.0, "hardware_profile": {}, "score": 1.0},
)
_post_json(
f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
{"endpoint": "http://127.0.0.1:9003", "model": "tiny-model",
"vram_bytes": 7_000, "ram_bytes": 10_000, "quantizations": ["bfloat16"],
"benchmark_tokens_per_sec": 1.0, "hardware_profile": {}, "score": 1.0},
)
route_resp = _get_json(f"http://127.0.0.1:{tracker_port}/v1/route?model=tiny-model")
widths = {
node["endpoint"]: node["shard_end"] - node["shard_start"] + 1
for node in route_resp["nodes"]
}
assert widths["http://127.0.0.1:9003"] == 5
assert widths["http://127.0.0.1:9002"] == 4
assert widths["http://127.0.0.1:9001"] == 3
coverage_resp = _get_json(f"http://127.0.0.1:{tracker_port}/v1/coverage/tiny-model")
assert all(segment["node_count"] >= 1 for segment in coverage_resp["coverage"])
finally:
tracker.stop()
def test_tracker_speed_is_primary_when_both_nodes_can_cover_gap():
tracker = TrackerServer(model_presets={
"tiny-model": {
"total_layers": 12,
"bytes_per_layer": {"bfloat16": 1_000},
},
})
tracker_port = tracker.start()
try:
_post_json(
f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
{"endpoint": "http://127.0.0.1:9011", "model": "tiny-model",
"vram_bytes": 20_000, "ram_bytes": 10_000, "quantizations": ["bfloat16"],
"benchmark_tokens_per_sec": 1.0, "hardware_profile": {}, "score": 1.0},
)
_post_json(
f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
{"endpoint": "http://127.0.0.1:9012", "model": "tiny-model",
"vram_bytes": 15_000, "ram_bytes": 10_000, "quantizations": ["bfloat16"],
"benchmark_tokens_per_sec": 3.0, "hardware_profile": {}, "score": 1.0},
)
route_resp = _get_json(f"http://127.0.0.1:{tracker_port}/v1/route?model=tiny-model")
widths = {
node["endpoint"]: node["shard_end"] - node["shard_start"] + 1
for node in route_resp["nodes"]
}
assert widths["http://127.0.0.1:9012"] > widths["http://127.0.0.1:9011"]
finally:
tracker.stop()
def test_tracker_registration_directive_is_not_replayed_on_heartbeat():
tracker = TrackerServer(model_presets={
"tiny-model": {
"total_layers": 2,
"bytes_per_layer": {"bfloat16": 1_000},
},
})
tracker_port = tracker.start()
try:
reg = _post_json(
f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
{"endpoint": "http://127.0.0.1:9013", "model": "tiny-model",
"vram_bytes": 5_000, "ram_bytes": 10_000, "quantizations": ["bfloat16"],
"benchmark_tokens_per_sec": 1.0, "hardware_profile": {}, "score": 1.0},
)
assert reg["directive"]["action"] == "LOAD_SHARD"
hb = _post_json(f"http://127.0.0.1:{tracker_port}/v1/nodes/{reg['node_id']}/heartbeat", {})
assert hb == {}
finally:
tracker.stop()
def test_tracker_reassignment_emits_drop_before_load():
tracker = TrackerServer(model_presets={
"tiny-model": {
"total_layers": 4,
"bytes_per_layer": {"bfloat16": 1_000},
},
})
tracker_port = tracker.start()
try:
slow = _post_json(
f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
{"endpoint": "http://127.0.0.1:9015", "model": "tiny-model",
"vram_bytes": 10_000, "ram_bytes": 10_000, "quantizations": ["bfloat16"],
"benchmark_tokens_per_sec": 1.0, "hardware_profile": {}, "score": 1.0},
)
_post_json(
f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
{"endpoint": "http://127.0.0.1:9016", "model": "tiny-model",
"vram_bytes": 10_000, "ram_bytes": 10_000, "quantizations": ["bfloat16"],
"benchmark_tokens_per_sec": 3.0, "hardware_profile": {}, "score": 1.0},
)
hb = _post_json(f"http://127.0.0.1:{tracker_port}/v1/nodes/{slow['node_id']}/heartbeat", {})
assert [directive["action"] for directive in hb["directives"]] == ["DROP_SHARD", "LOAD_SHARD"]
finally:
tracker.stop()
def test_tracker_periodic_rebalance_purges_expired_nodes_without_requests():
tracker = TrackerServer(
heartbeat_timeout=0.05,
rebalance_interval=0.02,
model_presets={"tiny-model": {"total_layers": 1, "bytes_per_layer": {"bfloat16": 1_000}}},
)
tracker_port = tracker.start()
try:
reg = _post_json(
f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
{"endpoint": "http://127.0.0.1:9014", "model": "tiny-model",
"vram_bytes": 5_000, "ram_bytes": 10_000, "quantizations": ["bfloat16"],
"benchmark_tokens_per_sec": 1.0, "hardware_profile": {}, "score": 1.0},
)
assert reg["node_id"] in tracker._registry
time.sleep(0.12)
assert reg["node_id"] not in tracker._registry
finally:
tracker.stop()
def test_tracker_faster_node_receives_wider_range_when_capacity_ties():
tracker = TrackerServer(model_presets={
"tiny-model": {
"total_layers": 12,
"bytes_per_layer": {"bfloat16": 1_000},
},
})
tracker_port = tracker.start()
try:
_post_json(
f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
{"endpoint": "http://127.0.0.1:9001", "model": "tiny-model",
"vram_bytes": 20_000, "ram_bytes": 10_000, "quantizations": ["bfloat16"],
"benchmark_tokens_per_sec": 1.0, "hardware_profile": {}, "score": 1.0},
)
_post_json(
f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
{"endpoint": "http://127.0.0.1:9002", "model": "tiny-model",
"vram_bytes": 20_000, "ram_bytes": 10_000, "quantizations": ["bfloat16"],
"benchmark_tokens_per_sec": 2.0, "hardware_profile": {}, "score": 1.0},
)
route_resp = _get_json(f"http://127.0.0.1:{tracker_port}/v1/route?model=tiny-model")
widths = {
node["endpoint"]: node["shard_end"] - node["shard_start"] + 1
for node in route_resp["nodes"]
}
assert widths["http://127.0.0.1:9002"] > widths["http://127.0.0.1:9001"]
finally:
tracker.stop()
def test_tracker_rebalances_after_middle_range_node_timeout():
"""Killing a middle shard queues LOAD_SHARD and restores coverage."""
tracker = TrackerServer(
heartbeat_timeout=0.15,
model_presets={
"tiny-model": {
"total_layers": 12,
"bytes_per_layer": {"bfloat16": 1_000},
},
},
)
tracker_port = tracker.start()
ids: dict[str, str] = {}
try:
for port in (9001, 9002, 9003, 9004):
reg = _post_json(
f"http://127.0.0.1:{tracker_port}/v1/nodes/register",
{"endpoint": f"http://127.0.0.1:{port}", "model": "tiny-model",
"vram_bytes": 5_000, "ram_bytes": 10_000, "quantizations": ["bfloat16"],
"benchmark_tokens_per_sec": 1.0, "hardware_profile": {}, "score": 1.0},
)
ids[str(port)] = reg["node_id"]
for node_id in ids.values():
_post_json(f"http://127.0.0.1:{tracker_port}/v1/nodes/{node_id}/heartbeat", {})
time.sleep(0.10)
for port in ("9001", "9003", "9004"):
_post_json(f"http://127.0.0.1:{tracker_port}/v1/nodes/{ids[port]}/heartbeat", {})
time.sleep(0.10)
coverage_resp = _get_json(f"http://127.0.0.1:{tracker_port}/v1/coverage/tiny-model")
assert all(segment["node_count"] >= 1 for segment in coverage_resp["coverage"])
heartbeat_responses = [
_post_json(f"http://127.0.0.1:{tracker_port}/v1/nodes/{ids[port]}/heartbeat", {})
for port in ("9001", "9003", "9004")
]
directives = [
directive
for response in heartbeat_responses
for directive in response.get("directives", [])
]
assert any(directive["action"] == "LOAD_SHARD" for directive in directives)
finally:
tracker.stop()
def test_tracker_route_error_no_nodes(): def test_tracker_route_error_no_nodes():
"""Tracker returns 503 with clear error when the registry is empty.""" """Tracker returns 503 with clear error when the registry is empty."""
tracker = TrackerServer() tracker = TrackerServer()