gogo2/TODO_IMPROVEMENTS.md

# Cryptocurrency Trading System Improvements

## Overview
This document outlines necessary improvements to our cryptocurrency trading system to enhance performance, profitability, and monitoring capabilities.

## High Priority Tasks

### 1. GPU Utilization for Training
- [x] Fix GPU detection and utilization during training
  - [x] Debug why CUDA is detected but not utilized (check logs showing "Starting training on device: cpu")
  - [x] Ensure PyTorch correctly detects and uses available CUDA devices
  - [x] Add GPU memory monitoring during training
  - [x] Optimize batch sizes for GPU training

Implementation status:
- Added `setup_gpu()` function in `train_rl_with_realtime.py` to properly detect and configure GPU usage
- Added device parameter to DQNAgent to ensure models are created on the correct device
- Implemented mixed precision training for faster GPU-based training
- Added GPU memory monitoring and logging to TensorBoard

### 2. Trade Signal Rate Display
- [x] Add metrics to track and display trading frequency
  - [x] Implement counter for actions per second/minute/hour
  - [x] Add visualization to the chart showing trading frequency over time
  - [x] Create a moving average of trade signals to show trends
  - [x] Add dashboard section showing current and average trading rates

Implementation status:
- Added trade time tracking in `_add_trade_compat` function
- Added `calculate_trade_rate` method to `RealTimeChart` class
- Updated dashboard layout to display trade rates
- Added visualization of trade frequency in chart's bottom panel

### 3. Reward Function Optimization
- [x] Revise reward function to better balance profit and risk
  - [x] Increase transaction fee penalty for more realistic simulation
  - [x] Implement progressive rewards based on holding time
  - [x] Add penalty for frequent trading (to reduce noise)
  - [x] Scale rewards based on market volatility
  - [x] Implement risk-adjusted returns (Sharpe ratio) in reward calculation

Implementation status:
- Created `improved_reward_function.py` with `ImprovedRewardCalculator` class
- Implemented Sharpe ratio for risk-adjusted rewards
- Added frequency penalty for excessive trading
- Added holding time rewards for profitable positions
- Integrated with `EnhancedRLTradingEnvironment` class

### 4. Multi-timeframe Price Direction Prediction
- [ ] Extend CNN model to predict price direction for multiple timeframes
  - [ ] Modify CNN output to predict short, mid, and long-term price directions
  - [ ] Create data generation method for back-propagation using historical data
  - [ ] Implement real-time example generation for training
  - [ ] Feed direction predictions to RL agent as additional state information

## Medium Priority Tasks

### 5. Position Sizing Optimization
- [ ] Implement dynamic position sizing based on confidence and volatility
  - [ ] Add confidence score to model outputs
  - [ ] Scale position size based on prediction confidence
  - [ ] Implement Kelly criterion for optimal position sizing

### 6. Training Data Augmentation
- [ ] Implement data augmentation for more robust training
  - [ ] Simulate different market conditions
  - [ ] Add noise to training data
  - [ ] Generate synthetic data for rare market events

### 7. Model Interpretability
- [ ] Add visualization for model decision making
  - [ ] Implement feature importance analysis
  - [ ] Add attention visualization for key price patterns
  - [ ] Create explainable AI components

## Implementation Details

### Completed: Displaying Trade Rate
The trade rate display implementation has been completed in the `RealTimeChart` class:
```python
def calculate_trade_rate(self):
    """Calculate and return trading rate statistics based on recent trades"""
    if not hasattr(self, 'trade_times') or not self.trade_times:
        return {"per_second": 0, "per_minute": 0, "per_hour": 0}
    
    # Get current time
    now = datetime.now()
    
    # Calculate different time windows
    one_second_ago = now - timedelta(seconds=1)
    one_minute_ago = now - timedelta(minutes=1)
    one_hour_ago = now - timedelta(hours=1)
    
    # Count trades in different time windows
    trades_last_second = sum(1 for t in self.trade_times if t > one_second_ago)
    trades_last_minute = sum(1 for t in self.trade_times if t > one_minute_ago)
    trades_last_hour = sum(1 for t in self.trade_times if t > one_hour_ago)
    
    # Calculate rates
    return {
        "per_second": trades_last_second,
        "per_minute": trades_last_minute,
        "per_hour": trades_last_hour
    }
```

### Completed: Improved Reward Function
The improved reward function has been implemented in `improved_reward_function.py`:
```python
def calculate_reward(self, action, price_change, position_held_time=0, 
                     volatility=None, is_profitable=False):
    """
    Calculate the improved reward with risk adjustment
    """
    # Calculate trading fee
    fee = self.base_fee_rate
    
    # Calculate frequency penalty
    frequency_penalty = self._calculate_frequency_penalty()
    
    # Base reward calculation
    if action == 0:  # BUY
        # Small penalty for transaction plus frequency penalty
        reward = -fee - frequency_penalty
        
    elif action == 1:  # SELL
        # Calculate profit percentage minus fees (both entry and exit)
        profit_pct = price_change
        net_profit = profit_pct - (fee * 2)
        
        # Scale reward and apply frequency penalty
        reward = net_profit * 10  # Scale reward
        reward -= frequency_penalty
        
        # Record PnL for risk adjustment
        self.record_pnl(net_profit)
        
    else:  # HOLD
        # Small reward for holding a profitable position, small cost otherwise
        if is_profitable:
            reward = self._calculate_holding_reward(position_held_time, price_change)
        else:
            reward = -0.0001  # Very small negative reward
    
    # Apply risk adjustment if enabled
    if self.risk_adjusted:
        reward = self._calculate_risk_adjustment(reward)
        
    # Record this action for future frequency calculations
    self.record_trade(action=action)
    
    return reward
```

### Completed: GPU Optimization
Added GPU optimization in `train_rl_with_realtime.py`:
```python
def setup_gpu():
    """
    Configure GPU usage for PyTorch training
    
    Returns:
        tuple: (success, device, message)
    """
    try:
        if torch.cuda.is_available():
            gpu_count = torch.cuda.device_count()
            device_info = [torch.cuda.get_device_name(i) for i in range(gpu_count)]
            logger.info(f"Found {gpu_count} GPU(s): {', '.join(device_info)}")
            
            device = torch.device("cuda:0")
            
            # Test CUDA by creating a small tensor
            test_tensor = torch.tensor([1.0, 2.0, 3.0], device=device)
            
            # Enable mixed precision if supported
            if hasattr(torch.cuda, 'amp') and torch.cuda.is_bf16_supported():
                logger.info("BFloat16 is supported - enabling for faster training")
            
            return True, device, f"GPU enabled: {device_info}"
        else:
            return False, torch.device("cpu"), "GPU not available, using CPU"
    except Exception as e:
        return False, torch.device("cpu"), f"GPU setup failed: {str(e)}"
```

### CNN Price Direction Prediction (To be implemented)
```python
def generate_direction_examples(self, historical_data, timeframes=['1m', '1h', '1d']):
    """Generate price direction examples from historical data"""
    examples = []
    labels = []
    
    for tf in timeframes:
        df = historical_data[tf]
        for i in range(20, len(df) - 10):
            # Use window of 20 candles for input
            window = df.iloc[i-20:i]
            
            # Create labels for future price direction (next 5, 10, 20 candles)
            future_5 = df.iloc[i].close < df.iloc[i+5].close  # True if price goes up
            future_10 = df.iloc[i].close < df.iloc[i+10].close
            future_20 = df.iloc[i].close < df.iloc[min(i+20, len(df)-1)].close
            
            examples.append(window.values)
            labels.append([future_5, future_10, future_20])
    
    return np.array(examples), np.array(labels)
```

## Validation Plan
After implementing these improvements, we should validate the system with:
1. Backtesting on historical data 
2. Forward testing with small position sizes
3. A/B testing of different reward functions
4. Measuring the improvement in profitability and Sharpe ratio

## Progress Tracking
- Implementation started: June 2023
- GPU utilization fixed: July 2023
- Trade signal rate display implemented: July 2023
- Reward function optimized: July 2023
- CNN direction prediction added: To be completed
- Full system tested: To be completed