150 lines
5.8 KiB
Markdown
150 lines
5.8 KiB
Markdown
# Project Cleanup & Reorganization Plan
|
|
|
|
## Current Issues
|
|
1. **Code Duplication**: Multiple CNN models, RL agents, training scripts doing similar things
|
|
2. **Missing Methods**: Core functionality like `run()`, `start_websocket()` missing from classes
|
|
3. **Unclear Architecture**: No clean separation between components
|
|
4. **Hard to Maintain**: Scattered implementations make changes difficult
|
|
|
|
## New Clean Architecture
|
|
|
|
```
|
|
gogo2/
|
|
├── core/ # Core system components
|
|
│ ├── __init__.py
|
|
│ ├── data_provider.py # Multi-timeframe, multi-symbol data
|
|
│ ├── orchestrator.py # Main decision making module
|
|
│ └── config.py # Central configuration
|
|
├── models/ # AI/ML Models
|
|
│ ├── __init__.py
|
|
│ ├── cnn/ # CNN module
|
|
│ │ ├── __init__.py
|
|
│ │ ├── model.py # Single CNN implementation
|
|
│ │ ├── trainer.py # CNN training pipeline
|
|
│ │ └── predictor.py # CNN inference with confidence
|
|
│ └── rl/ # RL module
|
|
│ ├── __init__.py
|
|
│ ├── agent.py # Single RL agent implementation
|
|
│ ├── environment.py # Trading environment
|
|
│ └── trainer.py # RL training loop
|
|
├── trading/ # Trading execution
|
|
│ ├── __init__.py
|
|
│ ├── executor.py # Trade execution
|
|
│ ├── portfolio.py # Position/portfolio management
|
|
│ └── metrics.py # Performance tracking
|
|
├── web/ # Web interface
|
|
│ ├── __init__.py
|
|
│ ├── dashboard.py # Main dashboard
|
|
│ └── charts.py # Chart components
|
|
├── utils/ # Utilities
|
|
│ ├── __init__.py
|
|
│ ├── logger.py # Centralized logging
|
|
│ └── helpers.py # Common helpers
|
|
├── main.py # Single entry point
|
|
├── config.yaml # Configuration file
|
|
└── requirements.txt # Dependencies
|
|
```
|
|
|
|
## Core Goals
|
|
|
|
### 1. Data Provider (`core/data_provider.py`)
|
|
- **Multi-symbol support**: ETH/USDT, BTC/USDT (configurable)
|
|
- **Multi-timeframe**: 1m, 5m, 15m, 1h, 4h, 1d
|
|
- **Real-time streaming**: WebSocket integration
|
|
- **Historical data**: API integration for backtesting
|
|
- **Clean interface**: Simple methods for getting data
|
|
|
|
### 2. CNN Module (`models/cnn/`)
|
|
- **Single model implementation**: Remove duplicates
|
|
- **Timeframe-specific predictions**: Separate predictions per timeframe
|
|
- **Confidence scoring**: Each prediction includes confidence
|
|
- **Training pipeline**: Supervised learning with marked data (perfect moves)
|
|
|
|
### 3. RL Module (`models/rl/`)
|
|
- **Single agent**: Remove duplicate DQN implementations
|
|
- **Environment**: Clean trading simulation
|
|
- **Learning loop**: Evaluates trading actions and adapts
|
|
|
|
### 4. Orchestrator (`core/orchestrator.py`)
|
|
- **Decision making**: Combines CNN and RL outputs
|
|
- **Final actions**: BUY/SELL/HOLD decisions
|
|
- **Confidence weighting**: Uses CNN confidence in decisions
|
|
|
|
### 5. Web Interface (`web/`)
|
|
- **Real-time charts**: Live trading visualization
|
|
- **Performance dashboard**: Metrics and analytics
|
|
- **Simple & clean**: Remove complex chart implementations
|
|
|
|
## Cleanup Steps
|
|
|
|
### Phase 1: Core Infrastructure
|
|
1. Create new clean directory structure
|
|
2. Implement `core/data_provider.py` (consolidate all data functionality)
|
|
3. Implement `core/orchestrator.py` (main decision maker)
|
|
4. Create `config.yaml` for all settings
|
|
|
|
### Phase 2: Model Consolidation
|
|
1. Create single `models/cnn/model.py` (consolidate all CNN implementations)
|
|
2. Create single `models/rl/agent.py` (consolidate DQN implementations)
|
|
3. Remove duplicate model files
|
|
|
|
### Phase 3: Training Simplification
|
|
1. Create `models/cnn/trainer.py` (single CNN training script)
|
|
2. Create `models/rl/trainer.py` (single RL training script)
|
|
3. Remove all duplicate training scripts
|
|
|
|
### Phase 4: Web Interface
|
|
1. Create clean `web/dashboard.py` (consolidate chart functionality)
|
|
2. Remove complex/unused chart implementations
|
|
|
|
### Phase 5: Integration & Testing
|
|
1. Create single `main.py` entry point
|
|
2. Test all components work together
|
|
3. Remove unused files
|
|
|
|
## Files to Remove (After consolidation)
|
|
|
|
### Duplicate Training Scripts
|
|
- `train_hybrid.py`
|
|
- `train_dqn.py`
|
|
- `train_cnn_with_realtime.py`
|
|
- `train_with_realtime_ticks.py`
|
|
- `train_improved_rl.py`
|
|
- `NN/train_enhanced.py`
|
|
- `NN/train_rl.py`
|
|
|
|
### Duplicate Model Files
|
|
- `NN/models/cnn_model.py`
|
|
- `NN/models/enhanced_cnn.py`
|
|
- `NN/models/simple_cnn.py`
|
|
- `NN/models/transformer_model.py`
|
|
- `NN/models/transformer_model_pytorch.py`
|
|
- `NN/models/dqn_agent_enhanced.py`
|
|
|
|
### Duplicate Main Files
|
|
- `trading_main.py`
|
|
- `NN/main.py`
|
|
- `NN/realtime_main.py`
|
|
- `NN/realtime-main.py`
|
|
|
|
### Unused Utilities
|
|
- `launch_training.py`
|
|
- `NN/example.py`
|
|
- Most logs and backup directories
|
|
|
|
## Benefits of New Architecture
|
|
|
|
1. **Single Source of Truth**: One implementation per component
|
|
2. **Clear Separation**: CNN, RL, and Orchestrator are distinct
|
|
3. **Easy to Extend**: Adding new symbols/timeframes is simple
|
|
4. **Maintainable**: Changes are localized to specific modules
|
|
5. **Testable**: Each component can be tested independently
|
|
|
|
## Implementation Priority
|
|
|
|
1. **HIGH**: Core data provider and orchestrator
|
|
2. **HIGH**: Single CNN and RL implementations
|
|
3. **MEDIUM**: Web dashboard consolidation
|
|
4. **LOW**: Cleanup of unused files
|
|
|
|
This plan will result in a much cleaner, more maintainable codebase focused on the core goal: multi-modal trading system with CNN predictions and RL decision making. |