232 lines
6.1 KiB
Markdown
232 lines
6.1 KiB
Markdown
# Clean Trading System
|
|
|
|
A modular, scalable cryptocurrency trading system with CNN and RL components for multi-timeframe analysis.
|
|
|
|
## 🚫 **CRITICAL: NO SYNTHETIC DATA POLICY**
|
|
|
|
**This system uses EXCLUSIVELY real market data from cryptocurrency exchanges. NO synthetic, generated, or simulated data is allowed for training, testing, or inference.**
|
|
|
|
### **Strictly Forbidden:**
|
|
- Any form of synthetic or generated data
|
|
- Mock or simulated market conditions
|
|
- Dummy data for testing or development
|
|
- Random price generation or manipulation
|
|
|
|
### **Policy Compliance:**
|
|
- All data must come from live exchange APIs
|
|
- Historical data must be authentic exchange records
|
|
- Real-time feeds must be direct from exchange WebSockets
|
|
- Zero tolerance for synthetic data in any form
|
|
|
|
**See `REAL_MARKET_DATA_POLICY.md` for complete details and compliance guidelines.**
|
|
|
|
## Features
|
|
|
|
- **Multi-timeframe Analysis**: 1s, 1m, 5m, 1h, 4h, 1d scalping focus
|
|
- **CNN Pattern Recognition**: Real market pattern detection with temporal attention
|
|
- **RL Trading Agent**: Reinforcement learning with real historical backtesting
|
|
- **Real-time Data**: Live market data from Binance API
|
|
- **Web Dashboard**: Real-time monitoring and visualization
|
|
- **Modular Architecture**: Clean separation of concerns
|
|
|
|
## Quick Start
|
|
|
|
### 1. Install Dependencies
|
|
```bash
|
|
pip install -r requirements.txt
|
|
```
|
|
|
|
### 2. Configure Settings
|
|
Edit `config.yaml` to set your preferences:
|
|
```yaml
|
|
symbols: ["ETH/USDT", "BTC/USDT"]
|
|
timeframes: ["1s", "1m", "5m", "1h", "4h"]
|
|
training:
|
|
use_only_real_data: true # CRITICAL: Never change this
|
|
```
|
|
|
|
### 3. Launch Real-time Dashboard & Training
|
|
```bash
|
|
# Full system with live training
|
|
python main_dashboard.py --port 8051
|
|
|
|
# Dashboard only (no training)
|
|
python main_dashboard.py --port 8051 --no-training
|
|
```
|
|
|
|
### 4. Run Backtesting & Bulk Training
|
|
```bash
|
|
# 30-day backtest
|
|
python main_backtest.py --start 2024-01-01 --end 2024-01-31
|
|
|
|
# Custom symbol and window
|
|
python main_backtest.py --start 2024-01-01 --end 2024-12-31 --symbol BTC/USDT --window 48
|
|
```
|
|
|
|
### 5. Unified Training Runner
|
|
```bash
|
|
# Realtime training for 4 hours
|
|
python training_runner.py --mode realtime --duration 4
|
|
|
|
# Backtest training
|
|
python training_runner.py --mode backtest --start-date 2024-01-01 --end-date 2024-12-31
|
|
```
|
|
|
|
## Architecture
|
|
|
|
```
|
|
gogo2/
|
|
├── core/ # Core system components
|
|
│ ├── config.py # Configuration management
|
|
│ ├── data_provider.py # Real market data fetching
|
|
│ └── orchestrator.py # Decision coordination
|
|
├── models/ # AI models (real data only)
|
|
│ ├── cnn/ # CNN pattern recognition
|
|
│ └── rl/ # RL trading agent
|
|
├── training/ # Training pipelines
|
|
│ ├── cnn_trainer.py # CNN training with real data
|
|
│ └── rl_trainer.py # RL training with real data
|
|
├── web/ # Web dashboard
|
|
└── main_clean.py # Unified entry point
|
|
```
|
|
|
|
## Data Sources
|
|
|
|
### Approved Sources
|
|
- Binance API (real-time and historical)
|
|
- Cached real market data
|
|
- TimescaleDB with real data
|
|
|
|
### Prohibited Sources
|
|
- Synthetic data generation
|
|
- Random data simulation
|
|
- Mock market conditions
|
|
|
|
## Training Modes
|
|
|
|
### CNN Training with TensorBoard
|
|
```bash
|
|
# Train on real ETH/USDT data with TensorBoard monitoring
|
|
python main_clean.py --mode cnn --symbol ETH/USDT
|
|
|
|
# Monitor training in real-time
|
|
tensorboard --logdir=runs
|
|
|
|
# Or use the convenience script
|
|
python run_tensorboard.py
|
|
|
|
# Quick test with real data
|
|
python test_cnn_only.py
|
|
```
|
|
|
|
### RL Training with TensorBoard
|
|
```bash
|
|
# Train RL agent with real data
|
|
python main_clean.py --mode rl --symbol ETH/USDT
|
|
|
|
# Real-time RL training
|
|
python train_rl_with_realtime.py --episodes 10
|
|
|
|
# Monitor RL training metrics
|
|
tensorboard --logdir=runs
|
|
```
|
|
|
|
## TensorBoard Monitoring
|
|
|
|
All training sessions are logged to TensorBoard for real-time monitoring:
|
|
|
|
```bash
|
|
# Start TensorBoard server
|
|
tensorboard --logdir=runs
|
|
|
|
# Or use the convenience script
|
|
python run_tensorboard.py
|
|
```
|
|
|
|
**Metrics Available:**
|
|
- **CNN Training**: Loss, accuracy, confidence scores, feature statistics
|
|
- **RL Training**: Rewards, returns, win rates, epsilon values, trading metrics
|
|
- **Model Architecture**: Parameter counts, memory usage
|
|
- **Real-time Updates**: Batch-level and epoch-level metrics
|
|
|
|
Access TensorBoard at: http://localhost:6006
|
|
|
|
## Performance
|
|
|
|
- **Memory Usage**: <2GB per model
|
|
- **Training Speed**: ~20 seconds for 50 epochs
|
|
- **Real Data Processing**: 1000+ candles per timeframe
|
|
- **Feature Count**: Dynamically detected from real data (typically 48)
|
|
|
|
## Monitoring
|
|
|
|
All operations log their data sources:
|
|
```
|
|
INFO - Generating 10000 training cases for ETH/USDT from REAL market data
|
|
INFO - Loaded 1000 real candles for ETH/USDT 1s
|
|
INFO - Building network with 48 features from real market data
|
|
```
|
|
|
|
## Testing
|
|
|
|
```bash
|
|
# Test data provider with real data
|
|
python -m pytest tests/test_data_provider.py
|
|
|
|
# Test CNN with real data
|
|
python test_cnn_only.py
|
|
|
|
# Test full system
|
|
python main_clean.py --mode test
|
|
```
|
|
|
|
## Web Dashboard
|
|
|
|
Access at `http://localhost:8050` for:
|
|
- Real-time price charts
|
|
- Model predictions
|
|
- Trading performance
|
|
- System metrics
|
|
|
|
## Configuration
|
|
|
|
Key settings in `config.yaml`:
|
|
```yaml
|
|
data:
|
|
provider: "binance" # Real exchange API
|
|
cache_enabled: true # Cache real data
|
|
real_time_enabled: true # Live data feed
|
|
|
|
training:
|
|
use_only_real_data: true # NEVER change this
|
|
batch_size: 32
|
|
epochs: 100
|
|
|
|
trading:
|
|
max_position_size: 0.1
|
|
trading_fee: 0.0002
|
|
```
|
|
|
|
## Safety Features
|
|
|
|
- **Data Validation**: Ensures all data comes from real sources
|
|
- **Cache Verification**: Validates cached data authenticity
|
|
- **Training Monitoring**: Logs all data sources
|
|
- **Emergency Stops**: Halts training if synthetic data detected
|
|
|
|
## Contributing
|
|
|
|
When contributing:
|
|
1. **NEVER** introduce synthetic data generation
|
|
2. Always use real market data for testing
|
|
3. Log data sources clearly
|
|
4. Follow the real data policy strictly
|
|
|
|
## License
|
|
|
|
This project is for educational and research purposes. Use real market data responsibly.
|
|
|
|
---
|
|
|
|
** REMEMBER: This system's integrity depends on using only real market data. No exceptions.**
|