inference working
This commit is contained in:
Dobromir Popov
@@ -0,0 +1,206 @@
|
||||
# US-016 — Mining-style node startup CLI + live dashboard
|
||||
|
||||
## Goal
|
||||
|
||||
Replace the bare flag-driven `meshnet-node start` with a wizard-guided first-run experience modelled on GPU mining clients (like PhoenixMiner, lolMiner, etc.). After the wizard, the terminal switches to a live status dashboard showing real-time node health and earnings.
|
||||
|
||||
## Wizard flow (first run only)
|
||||
|
||||
```
|
||||
╔══════════════════════════════════════════════════════════╗
|
||||
║ meshnet-node v0.1.0 ║
|
||||
║ Distributed AI Inference — Node Setup ║
|
||||
╚══════════════════════════════════════════════════════════╝
|
||||
|
||||
Detecting hardware...
|
||||
GPU 0: NVIDIA RTX 4090 24 GB VRAM ✓
|
||||
GPU 1: NVIDIA RTX 3090 24 GB VRAM ✓
|
||||
|
||||
Select a model to serve:
|
||||
|
||||
# Model Layers NF4 INT8 BF16
|
||||
1 Llama-3-70B-Instruct 80 ✓18GB ✓40GB ✗80GB
|
||||
2 Qwen-2.5-72B-Instruct 80 ✓19GB ✗41GB ✗81GB
|
||||
3 Mixtral-8x7B-Instruct-v0.1 32 ✓ 7GB ✓14GB ✓27GB
|
||||
4 Phi-3-medium-128k-instruct 40 ✓ 4GB ✓ 8GB ✓15GB
|
||||
5 [Browse HuggingFace…]
|
||||
|
||||
Enter number [1]: _
|
||||
|
||||
Quantization [nf4/int8/bf16] (nf4 recommended for 24GB): _
|
||||
|
||||
Download directory [~/.meshnet/models]: _
|
||||
|
||||
Tracker URL [http://localhost:8080]: _
|
||||
|
||||
Wallet path [~/.config/meshnet/wallet.json] (new wallet will be created): _
|
||||
|
||||
Config saved to ~/.config/meshnet/config.json
|
||||
Starting node…
|
||||
```
|
||||
|
||||
Second run with existing config:
|
||||
|
||||
```
|
||||
meshnet-node
|
||||
Reading config from ~/.config/meshnet/config.json
|
||||
Model: Llama-3-70B-Instruct Quant: nf4 Shard: layers 0–15
|
||||
Tracker: http://192.168.1.10:8080
|
||||
Starting…
|
||||
```
|
||||
|
||||
## Live dashboard (once running)
|
||||
|
||||
Renders every 2 seconds using `rich.live`. Fallback: plain-text status line if `rich` is unavailable or terminal is not a TTY (important for WSL2 / SSH).
|
||||
|
||||
```
|
||||
meshnet-node Llama-3-70B-Instruct [nf4] shard 0–15/80 up 00:03:22
|
||||
|
||||
GPU 0 RTX 4090 GPU ████████░░ 73% VRAM 18.2/24.0 GB 45°C
|
||||
GPU 1 RTX 3090 GPU ███░░░░░░░ 28% VRAM 8.7/24.0 GB 38°C
|
||||
|
||||
Tokens/sec ▁▂▃▄▅▆▇█ 42.3 t/s (EMA 30s)
|
||||
Requests 1,247 served 3 active
|
||||
Peers 8 connected (tracker: ✓ relay: ✓)
|
||||
TAI earned 0.00 TAI (payments active after US-006)
|
||||
Uptime 00:03:22
|
||||
|
||||
[q] quit [r] reset stats [c] compact view
|
||||
```
|
||||
|
||||
Compact mode (`--compact` or pressing `c`) shows a single status line:
|
||||
```
|
||||
[43t/s VRAM18.2GB req1247 peers8 up3m22s]
|
||||
```
|
||||
|
||||
## Implementation notes
|
||||
|
||||
### Hardware detection
|
||||
|
||||
```python
|
||||
import torch
|
||||
|
||||
def detect_gpus() -> list[dict]:
|
||||
gpus = []
|
||||
if torch.cuda.is_available():
|
||||
for i in range(torch.cuda.device_count()):
|
||||
props = torch.cuda.get_device_properties(i)
|
||||
gpus.append({
|
||||
"index": i,
|
||||
"name": props.name,
|
||||
"vram_gb": props.total_memory / 1e9,
|
||||
"backend": "cuda"
|
||||
})
|
||||
# ROCm / Apple Silicon stubs for later
|
||||
return gpus
|
||||
```
|
||||
|
||||
### Curated model list
|
||||
|
||||
`packages/node/meshnet_node/model_catalog.py` — a hardcoded list of `ModelPreset` dataclasses:
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class ModelPreset:
|
||||
name: str # display name
|
||||
hf_repo: str # HuggingFace repo ID
|
||||
num_layers: int
|
||||
vram_gb: dict # {"nf4": 18, "int8": 40, "bf16": 80}
|
||||
description: str # one-line description
|
||||
```
|
||||
|
||||
Initial list (expand over time):
|
||||
- `meta-llama/Meta-Llama-3-70B-Instruct` — 80L, NF4 18GB, INT8 40GB, BF16 80GB
|
||||
- `Qwen/Qwen2.5-72B-Instruct` — 80L, NF4 19GB, INT8 41GB, BF16 81GB
|
||||
- `mistralai/Mixtral-8x7B-Instruct-v0.1` — 32L, NF4 7GB, INT8 14GB, BF16 27GB
|
||||
- `microsoft/Phi-3-medium-128k-instruct` — 40L, NF4 4GB, INT8 8GB, BF16 15GB
|
||||
- `google/gemma-2-27b-it` — 46L, NF4 10GB, INT8 20GB, BF16 40GB
|
||||
|
||||
### HuggingFace Browse
|
||||
|
||||
```python
|
||||
from huggingface_hub import list_models
|
||||
|
||||
def browse_hf(top_n=20) -> list[dict]:
|
||||
models = list_models(
|
||||
pipeline_tag="text-generation",
|
||||
library="transformers",
|
||||
sort="downloads",
|
||||
direction=-1,
|
||||
limit=top_n,
|
||||
cardData=True,
|
||||
)
|
||||
return [{"repo": m.modelId, "downloads": m.downloads} for m in models]
|
||||
```
|
||||
|
||||
### Persistent config
|
||||
|
||||
`~/.config/meshnet/config.json`:
|
||||
```json
|
||||
{
|
||||
"model_hf_repo": "meta-llama/Meta-Llama-3-70B-Instruct",
|
||||
"quantization": "nf4",
|
||||
"download_dir": "~/.meshnet/models",
|
||||
"tracker_url": "http://192.168.1.10:8080",
|
||||
"wallet_path": "~/.config/meshnet/wallet.json",
|
||||
"shard_start": null,
|
||||
"shard_end": null,
|
||||
"updatedAt": "2026-06-29T..."
|
||||
}
|
||||
```
|
||||
|
||||
`shard_start`/`shard_end`: null means tracker auto-assigns. User can pin a range for dedicated partial-model nodes.
|
||||
|
||||
### CLI flags
|
||||
|
||||
All wizard answers are overridable without re-running the wizard:
|
||||
|
||||
```
|
||||
meshnet-node [start]
|
||||
--model <hf-repo-id> # e.g. meta-llama/Meta-Llama-3-70B-Instruct
|
||||
--quantization [bf16|int8|nf4]
|
||||
--download-dir <path>
|
||||
--tracker <url>
|
||||
--wallet <path>
|
||||
--shard-start <int> # pin shard range (optional)
|
||||
--shard-end <int>
|
||||
--reset-config # ignore saved config, re-run wizard
|
||||
--no-tui # plain-text output (for CI / headless)
|
||||
--compact # single-line status instead of full dashboard
|
||||
|
||||
meshnet-node models # list curated models and exit
|
||||
meshnet-node models --browse # list HF Hub top-20 and exit
|
||||
meshnet-node config # print current config and exit
|
||||
```
|
||||
|
||||
### WSL2 / non-TTY fallback
|
||||
|
||||
```python
|
||||
import sys, os
|
||||
|
||||
def is_interactive_tty() -> bool:
|
||||
return sys.stdout.isatty() and os.environ.get("TERM") not in ("dumb", "")
|
||||
|
||||
if not is_interactive_tty():
|
||||
# fall back to plain-text periodic status
|
||||
run_plain_status_loop(node)
|
||||
else:
|
||||
run_rich_dashboard(node)
|
||||
```
|
||||
|
||||
Do NOT use `termios`, `fcntl`, or `/dev/tty` — these break in Windows cmd.exe and some WSL2 terminal emulators.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- `meshnet-node` with no args and no config → wizard starts
|
||||
- Wizard detects GPU and marks `[too large]` for models that exceed available VRAM
|
||||
- `meshnet-node models` prints curated list and exits
|
||||
- `meshnet-node models --browse` calls HF Hub API, prints top-20, exits
|
||||
- Second run (config exists) → skips wizard, starts immediately
|
||||
- `--reset-config` re-runs wizard even with config present
|
||||
- All wizard inputs override-able via CLI flags
|
||||
- Live rich dashboard renders and updates every 2s when running in a TTY
|
||||
- Falls back to plain-text when not a TTY (CI / WSL2 without TERM set)
|
||||
- Ctrl-C prints a clean summary line and exits 0
|
||||
- `python -m pytest` passes from repo root
|
||||
- Commit only this story's changes
|
||||
Reference in New Issue
Block a user