182 lines
4.5 KiB
Markdown
182 lines
4.5 KiB
Markdown
# Technology Stack
|
|
|
|
## Core Technologies
|
|
|
|
### Python Ecosystem
|
|
- **Python 3.x**: Primary language
|
|
- **PyTorch**: Deep learning framework (CPU/CUDA/DirectML support)
|
|
- **NumPy/Pandas**: Data manipulation and analysis
|
|
- **scikit-learn**: ML utilities and preprocessing
|
|
|
|
### Web & API
|
|
- **Dash/Plotly**: Interactive web dashboard
|
|
- **Flask**: ANNOTATE web UI
|
|
- **FastAPI**: COBY REST API
|
|
- **WebSockets**: Real-time data streaming
|
|
|
|
### Data Storage
|
|
- **DuckDB**: Primary data storage (time-series optimized)
|
|
- **SQLite**: Metadata and predictions database
|
|
- **Redis**: High-performance caching (COBY)
|
|
- **TimescaleDB**: Optional time-series storage (COBY)
|
|
|
|
### Exchange Integration
|
|
- **ccxt**: Multi-exchange API library
|
|
- **websocket-client**: Real-time market data
|
|
- **pybit**: Bybit-specific integration
|
|
|
|
### Monitoring & Logging
|
|
- **TensorBoard**: Training visualization
|
|
- **wandb**: Experiment tracking
|
|
- **structlog**: Structured logging (COBY)
|
|
|
|
## Hardware Acceleration
|
|
|
|
### GPU Support
|
|
- NVIDIA CUDA (via PyTorch CUDA builds)
|
|
- AMD DirectML (via onnxruntime-directml)
|
|
- CPU fallback (default PyTorch CPU build)
|
|
|
|
**Note**: PyTorch is NOT in requirements.txt to avoid pulling NVIDIA CUDA deps on AMD machines. Install manually based on hardware.
|
|
|
|
## Project Structure
|
|
|
|
```
|
|
gogo2/
|
|
├── core/ # Core trading system components
|
|
├── models/ # Trained model checkpoints
|
|
├── NN/ # Neural network models and training
|
|
├── COBY/ # Multi-exchange data aggregation
|
|
├── ANNOTATE/ # Manual annotation UI
|
|
├── web/ # Main dashboard
|
|
├── utils/ # Shared utilities
|
|
├── cache/ # Data caching
|
|
├── data/ # Databases and exports
|
|
├── logs/ # System logs
|
|
└── @checkpoints/ # Model checkpoints archive
|
|
```
|
|
|
|
## Configuration
|
|
|
|
- **config.yaml**: Main system configuration (exchanges, symbols, timeframes, trading params)
|
|
- **models.yml**: Model-specific settings (CNN, RL, training)
|
|
- **.env**: Sensitive credentials (API keys, database passwords)
|
|
|
|
## Common Commands
|
|
|
|
### Running the System
|
|
|
|
```bash
|
|
# Main dashboard with live training
|
|
python main_dashboard.py --port 8051
|
|
|
|
# Dashboard without training
|
|
python main_dashboard.py --port 8051 --no-training
|
|
|
|
# Clean dashboard (alternative)
|
|
python run_clean_dashboard.py
|
|
```
|
|
|
|
### Training
|
|
|
|
```bash
|
|
# Unified training runner - realtime mode
|
|
python training_runner.py --mode realtime --duration 4
|
|
|
|
# Backtest training
|
|
python training_runner.py --mode backtest --start-date 2024-01-01 --end-date 2024-12-31
|
|
|
|
# CNN training with TensorBoard
|
|
python main_clean.py --mode cnn --symbol ETH/USDT
|
|
tensorboard --logdir=runs
|
|
|
|
# RL training
|
|
python main_clean.py --mode rl --symbol ETH/USDT
|
|
```
|
|
|
|
### Backtesting
|
|
|
|
```bash
|
|
# 30-day backtest
|
|
python main_backtest.py --start 2024-01-01 --end 2024-01-31
|
|
|
|
# Custom symbol and window
|
|
python main_backtest.py --start 2024-01-01 --end 2024-12-31 --symbol BTC/USDT --window 48
|
|
```
|
|
|
|
### COBY System
|
|
|
|
```bash
|
|
# Start COBY data aggregation
|
|
python COBY/main.py --debug
|
|
|
|
# Access COBY dashboard: http://localhost:8080
|
|
# COBY API: http://localhost:8080/api/...
|
|
# COBY WebSocket: ws://localhost:8081/dashboard
|
|
```
|
|
|
|
### ANNOTATE System
|
|
|
|
```bash
|
|
# Start annotation UI
|
|
python ANNOTATE/web/app.py
|
|
|
|
# Access at: http://127.0.0.1:8051
|
|
```
|
|
|
|
### Testing
|
|
|
|
```bash
|
|
# Run tests
|
|
python -m pytest tests/
|
|
|
|
# Test specific components
|
|
python test_cnn_only.py
|
|
python test_training.py
|
|
python test_duckdb_storage.py
|
|
```
|
|
|
|
### Monitoring
|
|
|
|
```bash
|
|
# TensorBoard for training metrics
|
|
tensorboard --logdir=runs
|
|
# Access at: http://localhost:6006
|
|
|
|
# Check data stream status
|
|
python check_stream.py status
|
|
python check_stream.py ohlcv
|
|
python check_stream.py cob
|
|
```
|
|
|
|
## Development Tools
|
|
|
|
- **TensorBoard**: Training visualization (runs/ directory)
|
|
- **wandb**: Experiment tracking
|
|
- **pytest**: Testing framework
|
|
- **Git**: Version control
|
|
|
|
## Dependencies Management
|
|
|
|
```bash
|
|
# Install dependencies
|
|
pip install -r requirements.txt
|
|
|
|
# Install PyTorch (choose based on hardware)
|
|
# CPU-only:
|
|
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
|
|
|
|
# NVIDIA GPU (CUDA 12.1):
|
|
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
|
|
|
|
# AMD NPU:
|
|
pip install onnxruntime-directml onnx transformers optimum
|
|
```
|
|
|
|
## Performance Targets
|
|
|
|
- **Memory Usage**: <2GB per model, <28GB total system
|
|
- **Training Speed**: ~20 seconds for 50 epochs
|
|
- **Inference Latency**: <200ms per prediction
|
|
- **Real Data Processing**: 1000+ candles per timeframe
|