inference_enabled, cleanup

2025-08-04 14:24:39 +03:00
parent 29382ac0db
commit e223bc90e9
39 changed files with 315 additions and 90858 deletions
--- a/COB_DATA_IMPROVEMENTS_SUMMARY.md
+++ b/COB_DATA_IMPROVEMENTS_SUMMARY.md
@ -1,125 +0,0 @@
 # COB Data Improvements Summary
 ## ✅ **Completed Improvements**
 ### 1. Fixed DateTime Comparison Error
 - **Issue**: `'<=' not supported between instances of 'datetime.datetime' and 'float'`
 - **Fix**: Added proper timestamp handling in `_aggregate_cob_1s()` method
 - **Result**: COB aggregation now works without datetime errors
 ### 2. Added Multi-timeframe Imbalance Indicators
 - **Added Indicators**:
  - `imbalance_1s`: Current 1-second imbalance
  - `imbalance_5s`: 5-second weighted average imbalance
  - `imbalance_15s`: 15-second weighted average imbalance  
  - `imbalance_60s`: 60-second weighted average imbalance
 - **Calculation Method**: Volume-weighted average with fallback to simple average
 - **Storage**: Added to both main data structure and stats section
 ### 3. Enhanced COB Data Structure
 - **Price Bucketing**: $1 USD price buckets for better granularity
 - **Volume Tracking**: Separate bid/ask volume tracking
 - **Statistics**: Comprehensive stats including spread, mid-price, volume
 - **Imbalance Calculation**: Proper bid-ask imbalance: `(bid_vol - ask_vol) / total_vol`
 ### 4. Added COB Data Quality Monitoring
 - **New Method**: `get_cob_data_quality()`
 - **Metrics Tracked**:
  - Raw tick count and freshness
  - Aggregated data count and freshness
  - Latest imbalance indicators
  - Data freshness assessment (excellent/good/fair/stale/no_data)
  - Price bucket counts
 ### 5. Improved Error Handling
 - **Robust Timestamp Handling**: Supports both datetime and float timestamps
 - **Graceful Degradation**: Returns default values when calculations fail
 - **Comprehensive Logging**: Detailed error messages for debugging
 ## 📊 **Test Results**
 ### Mock Data Test Results:
 - **✅ COB Aggregation**: Successfully processes ticks and creates 1s aggregated data
 - **✅ Imbalance Calculation**: 
  - 1s imbalance: 0.1044 (from current tick)
  - Multi-timeframe: 0.0000 (needs more historical data)
 - **✅ Price Bucketing**: 6 buckets created (3 bid + 3 ask)
 - **✅ Volume Tracking**: 594.00 total volume calculated correctly
 - **✅ Quality Monitoring**: All metrics properly reported
 ### Real-time Data Status:
 - **⚠️ WebSocket Connection**: Connecting but not receiving data yet
 - **❌ COB Provider Error**: `MultiExchangeCOBProvider.__init__() got an unexpected keyword argument 'bucket_size_bps'`
 - **✅ Data Structure**: Ready to receive and process real COB data
 ## 🔧 **Current Issues**
 ### 1. COB Provider Initialization Error
 - **Error**: `bucket_size_bps` parameter not recognized
 - **Impact**: Real COB data not flowing through system
 - **Status**: Needs investigation of COB provider interface
 ### 2. WebSocket Data Flow
 - **Status**: WebSocket connects but no data received yet
 - **Possible Causes**: 
  - COB provider initialization failure
  - WebSocket callback not properly connected
  - Data format mismatch
 ## 📈 **Data Quality Indicators**
 ### Imbalance Indicators (Working):
 ```python
 {
    'imbalance_1s': 0.1044,    # Current 1s imbalance
    'imbalance_5s': 0.0000,    # 5s weighted average
    'imbalance_15s': 0.0000,   # 15s weighted average  
    'imbalance_60s': 0.0000,   # 60s weighted average
    'total_volume': 594.00,    # Total volume
    'bucket_count': 6          # Price buckets
 }
 ```
 ### Data Freshness Assessment:
 - **excellent**: Data < 5 seconds old
 - **good**: Data < 15 seconds old
 - **fair**: Data < 60 seconds old
 - **stale**: Data > 60 seconds old
 - **no_data**: No data available
 ## 🎯 **Next Steps**
 ### 1. Fix COB Provider Integration
 - Investigate `bucket_size_bps` parameter issue
 - Ensure proper COB provider initialization
 - Test real WebSocket data flow
 ### 2. Validate Real-time Imbalances
 - Test with live market data
 - Verify multi-timeframe calculations
 - Monitor data quality in production
 ### 3. Integration Testing
 - Test with trading models
 - Verify dashboard integration
 - Performance testing under load
 ## 🔍 **Usage Examples**
 ### Get COB Data Quality:
 ```python
 dp = DataProvider()
 quality = dp.get_cob_data_quality()
 print(f"ETH imbalance 1s: {quality['imbalance_indicators']['ETH/USDT']['imbalance_1s']}")
 ```
 ### Get Recent Aggregated Data:
 ```python
 recent_cob = dp.get_cob_1s_aggregated('ETH/USDT', count=10)
 for record in recent_cob:
    print(f"Time: {record['timestamp']}, Imbalance: {record['imbalance_1s']:.4f}")
 ```
 ## ✅ **Summary**
 The COB data improvements are **functionally complete** and **tested**. The imbalance calculation system works correctly with multi-timeframe indicators. The main remaining issue is the COB provider initialization error that prevents real-time data flow. Once this is resolved, the system will provide high-quality COB data with comprehensive imbalance indicators for trading models.
--- a/COMPREHENSIVE_TRAINING_SYSTEM_SUMMARY.md
+++ b/COMPREHENSIVE_TRAINING_SYSTEM_SUMMARY.md
@ -1,289 +0,0 @@
 # Comprehensive Training System Implementation Summary
 ## 🎯 **Overview**
 I've successfully implemented a comprehensive training system that focuses on **proper training pipeline design with storing backpropagation training data** for both CNN and RL models. The system enables **replay and re-training on the best/most profitable setups** with complete data validation and integrity checking.
 ## 🏗️ **System Architecture**
 ```
 ┌─────────────────────────────────────────────────────────────────┐
 │                    COMPREHENSIVE TRAINING SYSTEM                 │
 ├─────────────────────────────────────────────────────────────────┤
 │                                                                 │
 │  ┌─────────────────┐    ┌──────────────────┐    ┌─────────────┐ │
 │  │ Data Collection │───▶│ Training Storage │───▶│ Validation  │ │
 │  │   & Validation  │    │   & Integrity    │    │ & Outcomes  │ │
 │  └─────────────────┘    └──────────────────┘    └─────────────┘ │
 │           │                       │                      │      │
 │           ▼                       ▼                      ▼      │
 │  ┌─────────────────┐    ┌──────────────────┐    ┌─────────────┐ │
 │  │ CNN Training    │    │ RL Training      │    │ Integration │ │
 │  │ Pipeline        │    │ Pipeline         │    │ & Replay    │ │
 │  └─────────────────┘    └──────────────────┘    └─────────────┘ │
 │                                                                 │
 └─────────────────────────────────────────────────────────────────┘
 ```
 ## 📁 **Files Created**
 ### **Core Training System**
 1. **`core/training_data_collector.py`** - Main data collection with validation
 2. **`core/cnn_training_pipeline.py`** - CNN training with backpropagation storage
 3. **`core/rl_training_pipeline.py`** - RL training with experience replay
 4. **`core/training_integration.py`** - Basic integration module
 5. **`core/enhanced_training_integration.py`** - Advanced integration with existing systems
 ### **Testing & Validation**
 6. **`test_training_data_collection.py`** - Individual component tests
 7. **`test_complete_training_system.py`** - Complete system integration test
 ## 🔥 **Key Features Implemented**
 ### **1. Comprehensive Data Collection & Validation**
 - **Data Integrity Hashing** - Every data package has MD5 hash for corruption detection
 - **Completeness Scoring** - 0.0 to 1.0 score with configurable minimum thresholds
 - **Validation Flags** - Multiple validation checks for data consistency
 - **Real-time Validation** - Continuous validation during collection
 ### **2. Profitable Setup Detection & Replay**
 - **Future Outcome Validation** - System knows which predictions were actually profitable
 - **Profitability Scoring** - Ranking system for all training episodes
 - **Training Priority Calculation** - Smart prioritization based on profitability and characteristics
 - **Selective Replay Training** - Train only on most profitable setups
 ### **3. Rapid Price Change Detection**
 - **Velocity-based Detection** - Detects % price change per minute
 - **Volatility Spike Detection** - Adaptive baseline with configurable multipliers
 - **Premium Training Examples** - Automatically collects high-value training data
 - **Configurable Thresholds** - Adjustable for different market conditions
 ### **4. Complete Backpropagation Data Storage**
 #### **CNN Training Pipeline:**
 - **CNNTrainingStep** - Stores every training step with:
  - Complete gradient information for all parameters
  - Loss component breakdown (classification, regression, confidence)
  - Model state snapshots at each step
  - Training value calculation for replay prioritization
 - **CNNTrainingSession** - Groups steps with profitability tracking
 - **Profitable Episode Replay** - Can retrain on most profitable pivot predictions
 #### **RL Training Pipeline:**
 - **RLExperience** - Complete state-action-reward-next_state storage with:
  - Actual trading outcomes and profitability metrics
  - Optimal action determination (what should have been done)
  - Experience value calculation for replay prioritization
 - **ProfitWeightedExperienceBuffer** - Advanced experience replay with:
  - Profit-weighted sampling for training
  - Priority calculation based on actual outcomes
  - Separate tracking of profitable vs unprofitable experiences
 - **RLTrainingStep** - Stores backpropagation data:
  - Complete gradient information
  - Q-value and policy loss components
  - Batch profitability metrics
 ### **5. Training Session Management**
 - **Session-based Training** - All training organized into sessions with metadata
 - **Training Value Scoring** - Each session gets value score for replay prioritization
 - **Convergence Tracking** - Monitors training progress and convergence
 - **Automatic Persistence** - All sessions saved to disk with metadata
 ### **6. Integration with Existing Systems**
 - **DataProvider Integration** - Seamless connection to your existing data provider
 - **COB RL Model Integration** - Works with your existing 1B parameter COB RL model
 - **Orchestrator Integration** - Connects with your orchestrator for decision making
 - **Real-time Processing** - Background workers for continuous operation
 ## 🎯 **How the System Works**
 ### **Data Collection Flow:**
 1. **Real-time Collection** - Continuously collects comprehensive market data packages
 2. **Data Validation** - Validates completeness and integrity of each package
 3. **Rapid Change Detection** - Identifies high-value training opportunities
 4. **Storage with Hashing** - Stores with integrity hashes and validation flags
 ### **Training Flow:**
 1. **Future Outcome Validation** - Determines which predictions were actually profitable
 2. **Priority Calculation** - Ranks all episodes/experiences by profitability and learning value
 3. **Selective Training** - Trains primarily on profitable setups
 4. **Gradient Storage** - Stores all backpropagation data for replay
 5. **Session Management** - Organizes training into valuable sessions for replay
 ### **Replay Flow:**
 1. **Profitability Analysis** - Identifies most profitable training episodes/experiences
 2. **Priority-based Selection** - Selects highest value training data
 3. **Gradient Replay** - Can replay exact training steps with stored gradients
 4. **Session Replay** - Can replay entire high-value training sessions
 ## 📊 **Data Validation & Completeness**
 ### **ModelInputPackage Validation:**
 ```python
@dataclass
 class ModelInputPackage:
    # Complete data package with validation
    data_hash: str = ""                    # MD5 hash for integrity
    completeness_score: float = 0.0        # 0.0 to 1.0 completeness
    validation_flags: Dict[str, bool]      # Multiple validation checks
    def _calculate_completeness(self) -> float:
        # Checks 10 required data fields
        # Returns percentage of complete fields
    def _validate_data(self) -> Dict[str, bool]:
        # Validates timestamp, OHLCV data, feature arrays
        # Checks data consistency and integrity
 ```
 ### **Training Outcome Validation:**
 ```python
@dataclass
 class TrainingOutcome:
    # Future outcome validation
    actual_profit: float                   # Real profit/loss
    profitability_score: float            # 0.0 to 1.0 profitability
    optimal_action: int                    # What should have been done
    is_profitable: bool                    # Binary profitability flag
    outcome_validated: bool = False        # Validation status
 ```
 ## 🔄 **Profitable Setup Replay System**
 ### **CNN Profitable Episode Replay:**
 ```python
 def train_on_profitable_episodes(self, 
                               symbol: str, 
                               min_profitability: float = 0.7,
                               max_episodes: int = 500):
    # 1. Get all episodes for symbol
    # 2. Filter for profitable episodes above threshold
    # 3. Sort by profitability score
    # 4. Train on most profitable episodes only
    # 5. Store all backpropagation data for future replay
 ```
 ### **RL Profit-Weighted Experience Replay:**
 ```python
 class ProfitWeightedExperienceBuffer:
    def sample_batch(self, batch_size: int, prioritize_profitable: bool = True):
        # 1. Sample mix of profitable and all experiences
        # 2. Weight sampling by profitability scores
        # 3. Prioritize experiences with positive outcomes
        # 4. Update training counts to avoid overfitting
 ```
 ## 🚀 **Ready for Production Integration**
 ### **Integration Points:**
 1. **Your DataProvider** - `enhanced_training_integration.py` ready to connect
 2. **Your CNN/RL Models** - Replace placeholder models with your actual ones
 3. **Your Orchestrator** - Integration hooks already implemented
 4. **Your Trading Executor** - Ready for outcome validation integration
 ### **Configuration:**
 ```python
 config = EnhancedTrainingConfig(
    collection_interval=1.0,              # Data collection frequency
    min_data_completeness=0.8,            # Minimum data quality threshold
    min_episodes_for_cnn_training=100,    # CNN training trigger
    min_experiences_for_rl_training=200,  # RL training trigger
    min_profitability_for_replay=0.1,     # Profitability threshold
    enable_background_validation=True,     # Real-time outcome validation
 )
 ```
 ## 🧪 **Testing & Validation**
 ### **Comprehensive Test Suite:**
 - **Individual Component Tests** - Each component tested in isolation
 - **Integration Tests** - Full system integration testing
 - **Data Integrity Tests** - Hash validation and completeness checking
 - **Profitability Replay Tests** - Profitable setup detection and replay
 - **Performance Tests** - Memory usage and processing speed validation
 ### **Test Results:**
 ```
 ✅ Data Collection: 100% integrity, 95% completeness average
 ✅ CNN Training: Profitable episode replay working, gradient storage complete
 ✅ RL Training: Profit-weighted replay working, experience prioritization active
 ✅ Integration: Real-time processing, outcome validation, cross-model learning
 ```
 ## 🎯 **Next Steps for Full Integration**
 ### **1. Connect to Your Infrastructure:**
 ```python
 # Replace mock with your actual DataProvider
 from core.data_provider import DataProvider
 data_provider = DataProvider(symbols=['ETH/USDT', 'BTC/USDT'])
 # Initialize with your components
 integration = EnhancedTrainingIntegration(
    data_provider=data_provider,
    orchestrator=your_orchestrator,
    trading_executor=your_trading_executor
 )
 ```
 ### **2. Replace Placeholder Models:**
 ```python
 # Use your actual CNN model
 your_cnn_model = YourCNNModel()
 cnn_trainer = CNNTrainer(your_cnn_model)
 # Use your actual RL model
 your_rl_agent = YourRLAgent()
 rl_trainer = RLTrainer(your_rl_agent)
 ```
 ### **3. Enable Real Outcome Validation:**
 ```python
 # Connect to live price feeds for outcome validation
 def _calculate_prediction_outcome(self, prediction_data):
    # Get actual price movements after prediction
    # Calculate real profitability
    # Update experience outcomes
 ```
 ### **4. Deploy with Monitoring:**
 ```python
 # Start the complete system
 integration.start_enhanced_integration()
 # Monitor performance
 stats = integration.get_integration_statistics()
 ```
 ## 🏆 **System Benefits**
 ### **For Training Quality:**
 - **Only train on profitable setups** - No wasted training on bad examples
 - **Complete gradient replay** - Can replay exact training steps
 - **Data integrity guaranteed** - Hash validation prevents corruption
 - **Rapid change detection** - Captures high-value training opportunities
 ### **For Model Performance:**
 - **Profit-weighted learning** - Models learn from successful examples
 - **Cross-model integration** - CNN and RL models share information
 - **Real-time validation** - Immediate feedback on prediction quality
 - **Adaptive prioritization** - Training focus shifts to most valuable data
 ### **For System Reliability:**
 - **Comprehensive validation** - Multiple layers of data checking
 - **Background processing** - Doesn't interfere with trading operations
 - **Automatic persistence** - All training data saved for replay
 - **Performance monitoring** - Real-time statistics and health checks
 ## 🎉 **Ready to Deploy!**
 The comprehensive training system is **production-ready** and designed to integrate seamlessly with your existing infrastructure. It provides:
 - ✅ **Complete data validation and integrity checking**
 - ✅ **Profitable setup detection and replay training**
 - ✅ **Full backpropagation data storage for gradient replay**
 - ✅ **Rapid price change detection for premium training examples**
 - ✅ **Real-time outcome validation and profitability tracking**
 - ✅ **Integration with your existing DataProvider and models**
 **The system is ready to start collecting training data and improving your models' performance through selective training on profitable setups!**
--- a/DATA_PROVIDER_CHANGES_SUMMARY.md
+++ b/DATA_PROVIDER_CHANGES_SUMMARY.md
@ -1,112 +0,0 @@
 # Data Provider Simplification Summary
 ## Changes Made
 ### 1. Removed Pre-loading System
 - Removed `_should_preload_data()` method
 - Removed `_preload_300s_data()` method  
 - Removed `preload_all_symbols_data()` method
 - Removed all pre-loading logic from `get_historical_data()`
 ### 2. Simplified Data Structure
 - Fixed symbols to `['ETH/USDT', 'BTC/USDT']`
 - Fixed timeframes to `['1s', '1m', '1h', '1d']`
 - Replaced `historical_data` with `cached_data` structure
 - Each symbol/timeframe maintains exactly 1500 OHLCV candles (limited by API to ~1000)
 ### 3. Automatic Data Maintenance System
 - Added `start_automatic_data_maintenance()` method
 - Added `_data_maintenance_worker()` background thread
 - Added `_initial_data_load()` for startup data loading
 - Added `_update_cached_data()` for periodic updates
 ### 4. Data Update Strategy
 - Initial load: Fetch 1500 candles for each symbol/timeframe at startup
 - Periodic updates: Fetch last 2 candles every half candle period
  - 1s data: Update every 0.5 seconds
  - 1m data: Update every 30 seconds  
  - 1h data: Update every 30 minutes
  - 1d data: Update every 12 hours
 ### 5. API Call Isolation
 - `get_historical_data()` now only returns cached data
 - No external API calls triggered by data requests
 - All API calls happen in background maintenance thread
 - Rate limiting increased to 500ms between requests
 ### 6. Updated Methods
 - `get_historical_data()`: Returns cached data only
 - `get_latest_candles()`: Uses cached data + real-time data
 - `get_current_price()`: Uses cached data only
 - `get_price_at_index()`: Uses cached data only
 - `get_feature_matrix()`: Uses cached data only
 - `_get_cached_ohlcv_bars()`: Simplified to use cached data
 - `health_check()`: Updated to show cached data status
 ### 7. New Methods Added
 - `get_cached_data_summary()`: Returns detailed cache status
 - `stop_automatic_data_maintenance()`: Stops background updates
 ### 8. Removed Methods
 - All pre-loading related methods
 - `invalidate_ohlcv_cache()` (no longer needed)
 - `_build_ohlcv_bar_cache()` (simplified)
 ## Test Results
 ### ✅ **Test Script Results:**
 - **Initial Data Load**: Successfully loaded 1000 candles for each symbol/timeframe
 - **Cached Data Access**: `get_historical_data()` returns cached data without API calls
 - **Current Price Retrieval**: Works correctly from cached data (ETH: $3,809, BTC: $118,290)
 - **Automatic Updates**: Background maintenance thread updating data every half candle period
 - **WebSocket Integration**: COB WebSocket connecting and working properly
 ### 📊 **Data Loaded:**
 - **ETH/USDT**: 1s, 1m, 1h, 1d (1000 candles each)
 - **BTC/USDT**: 1s, 1m, 1h, 1d (1000 candles each)
 - **Total**: 8,000 OHLCV candles cached and maintained automatically
 ### 🔧 **Minor Issues:**
 - Initial load gets ~1000 candles instead of 1500 (Binance API limit)
 - Some WebSocket warnings on Windows (non-critical)
 - COB provider initialization error (doesn't affect main functionality)
 ## Benefits
 1. **Predictable Performance**: No unexpected API calls during data requests
 2. **Rate Limit Compliance**: All API calls controlled in background thread
 3. **Consistent Data**: Always 1000+ candles available for each symbol/timeframe
 4. **Real-time Updates**: Data stays fresh with automatic background updates
 5. **Simplified Architecture**: Clear separation between data access and data fetching
 ## Usage
 ```python
 # Initialize data provider (starts automatic maintenance)
 dp = DataProvider()
 # Get cached data (no API calls)
 data = dp.get_historical_data('ETH/USDT', '1m', limit=100)
 # Get current price from cache
 price = dp.get_current_price('ETH/USDT')
 # Check cache status
 summary = dp.get_cached_data_summary()
 # Stop maintenance when done
 dp.stop_automatic_data_maintenance()
 ```
 ## Test Scripts
 - `test_simplified_data_provider.py`: Basic functionality test
 - `example_usage_simplified_data_provider.py`: Comprehensive usage examples
 ## Performance Metrics
 - **Startup Time**: ~15 seconds for initial data load
 - **Memory Usage**: ~8,000 OHLCV candles in memory
 - **API Calls**: Controlled background updates only
 - **Data Freshness**: Updated every half candle period
 - **Cache Hit Rate**: 100% for data requests (no API calls)
--- a/HOLD_POSITION_EVALUATION_FIX_SUMMARY.md
+++ b/HOLD_POSITION_EVALUATION_FIX_SUMMARY.md
@ -1,143 +0,0 @@
 # HOLD Position Evaluation Fix Summary
 ## Problem Description
 The trading system was incorrectly evaluating HOLD decisions without considering whether we're currently holding a position. This led to scenarios where:
 - HOLD was marked as incorrect even when price dropped while we were holding a profitable position
 - The system didn't differentiate between HOLD when we have a position vs. when we don't
 - Models weren't receiving position information as part of their input state
 ## Root Cause
 The issue was in the `_calculate_sophisticated_reward` method in `core/orchestrator.py`. The HOLD evaluation logic only considered price movement but ignored position status:
 ```python
 elif predicted_action == "HOLD":
    was_correct = abs(price_change_pct) < movement_threshold
    directional_accuracy = max(
        0, movement_threshold - abs(price_change_pct)
    )  # Positive for stability
 ```
 ## Solution Implemented
 ### 1. Enhanced Reward Calculation (`core/orchestrator.py`)
 Updated `_calculate_sophisticated_reward` method to:
 - Accept `symbol` and `has_position` parameters
 - Implement position-aware HOLD evaluation logic:
  - **With position**: HOLD is correct if price goes up (profit) or stays stable
  - **Without position**: HOLD is correct if price stays relatively stable
  - **With position + price drop**: Less penalty than wrong directional trades
 ```python
 elif predicted_action == "HOLD":
    # HOLD evaluation now considers position status
    if has_position:
        # If we have a position, HOLD is correct if price moved favorably or stayed stable
        if price_change_pct > 0:  # Price went up while holding - good
            was_correct = True
            directional_accuracy = price_change_pct  # Reward based on profit
        elif abs(price_change_pct) < movement_threshold:  # Price stable - neutral
            was_correct = True
            directional_accuracy = movement_threshold - abs(price_change_pct)
        else:  # Price dropped while holding - bad, but less penalty than wrong direction
            was_correct = False
            directional_accuracy = max(0, movement_threshold - abs(price_change_pct)) * 0.5
    else:
        # If we don't have a position, HOLD is correct if price stayed relatively stable
        was_correct = abs(price_change_pct) < movement_threshold
        directional_accuracy = max(
            0, movement_threshold - abs(price_change_pct)
        )  # Positive for stability
 ```
 ### 2. Enhanced BaseDataInput with Position Information (`core/data_models.py`)
 Added position information to the BaseDataInput class:
 - Added `position_info` field to store position state
 - Updated `get_feature_vector()` to include 5 position features:
  1. `has_position` (0.0 or 1.0)
  2. `position_pnl` (current P&L)
  3. `position_size` (position size)
  4. `entry_price` (entry price)
  5. `time_in_position_minutes` (time holding position)
 ### 3. Enhanced Orchestrator BaseDataInput Building (`core/orchestrator.py`)
 Updated `build_base_data_input` method to populate position information:
 - Retrieves current position status using `_has_open_position()`
 - Calculates position P&L using `_get_current_position_pnl()`
 - Gets detailed position information from trading executor
 - Adds all position data to `base_data.position_info`
 ### 4. Updated Method Calls
 Updated all calls to `_calculate_sophisticated_reward` to pass the new parameters:
 - Pass `symbol` for position lookup
 - Include fallback logic in exception handling
 ## Test Results
 The fix was validated with comprehensive tests:
 ### HOLD Evaluation Tests
 - ✅ HOLD with position + price up: CORRECT (making profit)
 - ✅ HOLD with position + price down: CORRECT (less penalty)
 - ✅ HOLD without position + small changes: CORRECT (avoiding unnecessary trades)
 ### Feature Integration Tests
 - ✅ BaseDataInput includes position_info with 5 features
 - ✅ Feature vector maintains correct size (7850 features)
 - ✅ CNN model successfully processes position information
 - ✅ Position features are correctly populated in feature vector
 ## Impact
 ### Immediate Benefits
 1. **Accurate HOLD Evaluation**: HOLD decisions are now evaluated correctly based on position status
 2. **Better Training Data**: Models receive more accurate reward signals for learning
 3. **Position-Aware Models**: All models now have access to current position information
 4. **Improved Decision Making**: Models can make better decisions knowing their position status
 ### Expected Improvements
 1. **Reduced False Negatives**: HOLD decisions won't be incorrectly penalized when holding profitable positions
 2. **Better Model Performance**: More accurate training signals should improve model accuracy over time
 3. **Context-Aware Trading**: Models can now consider position context when making decisions
 ## Files Modified
 1. **`core/orchestrator.py`**:
   - Enhanced `_calculate_sophisticated_reward()` method
   - Updated `build_base_data_input()` method
   - Updated method calls to pass position information
 2. **`core/data_models.py`**:
   - Added `position_info` field to BaseDataInput
   - Updated `get_feature_vector()` to include position features
   - Adjusted feature allocation (45 prediction features + 5 position features)
 3. **`test_hold_position_fix.py`** (new):
   - Comprehensive test suite to validate the fix
   - Tests HOLD evaluation with different position scenarios
   - Validates feature vector integration
 ## Backward Compatibility
 The changes are backward compatible:
 - Existing models will receive position information as additional features
 - Feature vector size remains 7850 (adjusted allocation internally)
 - All existing functionality continues to work as before
 ## Monitoring
 To monitor the effectiveness of this fix:
 1. Watch for improved HOLD decision accuracy in logs
 2. Monitor model training performance metrics
 3. Check that position information is correctly populated in feature vectors
 4. Observe overall trading system performance improvements
 ## Conclusion
 This fix addresses a critical issue in HOLD decision evaluation by making the system position-aware. The implementation is comprehensive, well-tested, and should lead to more accurate model training and better trading decisions.
--- a/MODEL_CLEANUP_SUMMARY.md
+++ b/MODEL_CLEANUP_SUMMARY.md
@ -1,137 +0,0 @@
 # Model Cleanup Summary Report
 *Completed: 2024-12-19*
 ## 🎯 Objective
 Clean up redundant and unused model implementations while preserving valuable architectural concepts and maintaining the production system integrity.
 ## 📋 Analysis Completed
 - **Comprehensive Analysis**: Created detailed report of all model implementations
 - **Good Ideas Documented**: Identified and recorded 50+ valuable architectural concepts
 - **Production Models Identified**: Confirmed which models are actively used
 - **Cleanup Plan Executed**: Removed redundant implementations systematically
 ## 🗑️ Files Removed
 ### CNN Model Implementations (4 files removed)
 - ✅ `NN/models/cnn_model_pytorch.py` - Superseded by enhanced version
 - ✅ `NN/models/enhanced_cnn_with_orderbook.py` - Functionality integrated elsewhere  
 - ✅ `NN/models/transformer_model_pytorch.py` - Basic implementation superseded
 - ✅ `training/williams_market_structure.py` - Fallback no longer needed
 ### Enhanced Training System (5 files removed)
 - ✅ `enhanced_rl_diagnostic.py` - Diagnostic script no longer needed
 - ✅ `enhanced_realtime_training.py` - Functionality integrated into orchestrator
 - ✅ `enhanced_rl_training_integration.py` - Superseded by orchestrator integration
 - ✅ `test_enhanced_training.py` - Test for removed functionality
 - ✅ `run_enhanced_cob_training.py` - Runner integrated into main system
 ### Test Files (3 files removed)
 - ✅ `tests/test_enhanced_rl_status.py` - Testing removed enhanced RL system
 - ✅ `tests/test_enhanced_dashboard_training.py` - Testing removed training system
 - ✅ `tests/test_enhanced_system.py` - Testing removed enhanced system
 ## ✅ Files Preserved (Production Models)
 ### Core Production Models
 - 🔒 `NN/models/cnn_model.py` - Main production CNN (Enhanced, 256+ channels)
 - 🔒 `NN/models/dqn_agent.py` - Main production DQN (Enhanced CNN backbone)
 - 🔒 `NN/models/cob_rl_model.py` - COB-specific RL (400M+ parameters)
 - 🔒 `core/nn_decision_fusion.py` - Neural decision fusion
 ### Advanced Architectures (Archived for Future Use)
 - 📦 `NN/models/advanced_transformer_trading.py` - 46M parameter transformer
 - 📦 `NN/models/enhanced_cnn.py` - Alternative CNN architecture
 - 📦 `NN/models/transformer_model.py` - MoE and transformer concepts
 ### Management Systems
 - 🔒 `model_manager.py` - Model lifecycle management
 - 🔒 `utils/checkpoint_manager.py` - Checkpoint management
 ## 🔄 Updates Made
 ### Import Updates
 - ✅ Updated `NN/models/__init__.py` to reflect removed files
 - ✅ Fixed imports to use correct remaining implementations
 - ✅ Added proper exports for production models
 ### Architecture Compliance
 - ✅ Maintained single source of truth for each model type
 - ✅ Preserved all good architectural ideas in documentation
 - ✅ Kept production system fully functional
 ## 💡 Good Ideas Preserved in Documentation
 ### Architecture Patterns
 1. **Multi-Scale Processing** - Multiple kernel sizes and attention scales
 2. **Attention Mechanisms** - Multi-head, self-attention, spatial attention
 3. **Residual Connections** - Pre-activation, enhanced residual blocks
 4. **Adaptive Architecture** - Dynamic network rebuilding
 5. **Normalization Strategies** - GroupNorm, LayerNorm for different scenarios
 ### Training Innovations
 1. **Experience Replay Variants** - Priority replay, example sifting
 2. **Mixed Precision Training** - GPU optimization and memory efficiency
 3. **Checkpoint Management** - Performance-based saving
 4. **Model Fusion** - Neural decision fusion, MoE architectures
 ### Market-Specific Features
 1. **Order Book Integration** - COB-specific preprocessing
 2. **Market Regime Detection** - Regime-aware models
 3. **Uncertainty Quantification** - Confidence estimation
 4. **Position Awareness** - Position-aware action selection
 ## 📊 Cleanup Statistics
 | Category | Files Analyzed | Files Removed | Files Preserved | Good Ideas Documented |
 |----------|----------------|---------------|-----------------|----------------------|
 | CNN Models | 5 | 4 | 1 | 12 |
 | Transformer Models | 3 | 1 | 2 | 8 |
 | RL Models | 2 | 0 | 2 | 6 |
 | Training Systems | 5 | 5 | 0 | 10 |
 | Test Files | 50+ | 3 | 47+ | - |
 | **Total** | **65+** | **13** | **52+** | **36** |
 ## 🎯 Results
 ### Space Saved
 - **Removed Files**: 13 files (~150KB of code)
 - **Reduced Complexity**: Eliminated 4 redundant CNN implementations
 - **Cleaner Architecture**: Single source of truth for each model type
 ### Knowledge Preserved
 - **Comprehensive Documentation**: All good ideas documented in detail
 - **Implementation Roadmap**: Clear path for future integrations
 - **Architecture Patterns**: Reusable patterns identified and documented
 ### Production System
 - **Zero Downtime**: All production models preserved and functional
 - **Enhanced Imports**: Cleaner import structure
 - **Future Ready**: Clear path for integrating documented innovations
 ## 🚀 Next Steps
 ### High Priority Integrations
 1. Multi-scale attention mechanisms → Main CNN
 2. Market regime detection → Orchestrator  
 3. Uncertainty quantification → Decision fusion
 4. Enhanced experience replay → Main DQN
 ### Medium Priority
 1. Relative positional encoding → Future transformer
 2. Advanced normalization strategies → All models
 3. Adaptive architecture features → Main models
 ### Future Considerations
 1. MoE architecture for ensemble learning
 2. Ultra-massive model variants for specialized tasks
 3. Advanced transformer integration when needed
 ## ✅ Conclusion
 Successfully cleaned up the project while:
 - **Preserving** all production functionality
 - **Documenting** valuable architectural innovations  
 - **Reducing** code complexity and redundancy
 - **Maintaining** clear upgrade paths for future enhancements
 The project is now cleaner, more maintainable, and ready for focused development on the core production models while having a clear roadmap for integrating the best ideas from the removed implementations. 
--- a/MODEL_IMPLEMENTATIONS_ANALYSIS_REPORT.md
+++ b/MODEL_IMPLEMENTATIONS_ANALYSIS_REPORT.md
@ -1,303 +0,0 @@
 # Model Implementations Analysis Report
 *Generated: 2024-12-19*
 ## Executive Summary
 This report analyzes all model implementations in the gogo2 trading system to identify valuable concepts and architectures before cleanup. The project contains multiple implementations of similar models, some unused, some experimental, and some production-ready.
 ## Current Model Ecosystem
 ### 🧠 CNN Models (5 Implementations)
 #### 1. **`NN/models/cnn_model.py`** - Production Enhanced CNN
 - **Status**: Currently used
 - **Architecture**: Ultra-massive 256+ channel architecture with 12+ residual blocks
 - **Key Features**:
  - Multi-head attention mechanisms (16 heads)
  - Multi-scale convolutional paths (3, 5, 7, 9 kernels)
  - Spatial attention blocks
  - GroupNorm for batch_size=1 compatibility
  - Memory barriers to prevent in-place operations
  - 2-action system optimized (BUY/SELL)
 - **Good Ideas**:
  - ✅ Attention mechanisms for temporal relationships
  - ✅ Multi-scale feature extraction
  - ✅ Robust normalization for single-sample inference
  - ✅ Memory management for gradient computation
  - ✅ Modular residual architecture
 #### 2. **`NN/models/enhanced_cnn.py`** - Alternative Enhanced CNN
 - **Status**: Alternative implementation
 - **Architecture**: Ultra-massive with 3072+ channels, deep residual blocks
 - **Key Features**:
  - Self-attention mechanisms
  - Pre-activation residual blocks
  - Ultra-massive fully connected layers (3072 → 2560 → 2048 → 1536 → 1024)
  - Adaptive network rebuilding based on input
  - Example sifting dataset for experience replay
 - **Good Ideas**:
  - ✅ Pre-activation residual design
  - ✅ Adaptive architecture based on input shape
  - ✅ Experience replay integration in CNN training
  - ✅ Ultra-wide hidden layers for complex pattern learning
 #### 3. **`NN/models/cnn_model_pytorch.py`** - Standard PyTorch CNN
 - **Status**: Standard implementation
 - **Architecture**: Standard CNN with basic features
 - **Good Ideas**:
  - ✅ Clean PyTorch implementation patterns
  - ✅ Standard training loops
 #### 4. **`NN/models/enhanced_cnn_with_orderbook.py`** - COB-Specific CNN
 - **Status**: Specialized for order book data
 - **Good Ideas**:
  - ✅ Order book specific preprocessing
  - ✅ Market microstructure awareness
 #### 5. **`training/williams_market_structure.py`** - Fallback CNN
 - **Status**: Fallback implementation
 - **Good Ideas**:
  - ✅ Graceful fallback mechanism
  - ✅ Simple architecture for testing
 ### 🤖 Transformer Models (3 Implementations)
 #### 1. **`NN/models/transformer_model.py`** - TensorFlow Transformer
 - **Status**: TensorFlow-based (outdated)
 - **Architecture**: Classic transformer with positional encoding
 - **Key Features**:
  - Multi-head attention
  - Positional encoding
  - Mixture of Experts (MoE) model
  - Time series + feature input combination
 - **Good Ideas**:
  - ✅ Positional encoding for temporal data
  - ✅ MoE architecture for ensemble learning
  - ✅ Multi-input design (time series + features)
  - ✅ Configurable attention heads and layers
 #### 2. **`NN/models/transformer_model_pytorch.py`** - PyTorch Transformer
 - **Status**: PyTorch migration
 - **Good Ideas**:
  - ✅ PyTorch implementation patterns
  - ✅ Modern transformer architecture
 #### 3. **`NN/models/advanced_transformer_trading.py`** - Advanced Trading Transformer
 - **Status**: Highly specialized
 - **Architecture**: 46M parameter transformer with advanced features
 - **Key Features**:
  - Relative positional encoding
  - Deep multi-scale attention (scales: 1,3,5,7,11,15)
  - Market regime detection
  - Uncertainty estimation
  - Enhanced residual connections
  - Layer norm variants
 - **Good Ideas**:
  - ✅ Relative positional encoding for temporal relationships
  - ✅ Multi-scale attention for different time horizons
  - ✅ Market regime detection integration
  - ✅ Uncertainty quantification
  - ✅ Deep attention mechanisms
  - ✅ Cross-scale attention
  - ✅ Market-specific configuration dataclass
 ### 🎯 RL Models (2 Implementations)
 #### 1. **`NN/models/dqn_agent.py`** - Enhanced DQN Agent
 - **Status**: Production system
 - **Architecture**: Enhanced CNN backbone with DQN
 - **Key Features**:
  - Priority experience replay
  - Checkpoint management integration
  - Mixed precision training
  - Position management awareness
  - Extrema detection integration
  - GPU optimization
 - **Good Ideas**:
  - ✅ Enhanced CNN as function approximator
  - ✅ Priority experience replay
  - ✅ Checkpoint management
  - ✅ Mixed precision for performance
  - ✅ Market context awareness
  - ✅ Position-aware action selection
 #### 2. **`NN/models/cob_rl_model.py`** - COB-Specific RL
 - **Status**: Specialized for order book
 - **Architecture**: Massive RL network (400M+ parameters)
 - **Key Features**:
  - Ultra-massive architecture for complex patterns
  - COB-specific preprocessing
  - Mixed precision training
  - Model interface for easy integration
 - **Good Ideas**:
  - ✅ Massive capacity for complex market patterns
  - ✅ COB-specific design
  - ✅ Interface pattern for model management
  - ✅ Mixed precision optimization
 ### 🔗 Decision Fusion Models
 #### 1. **`core/nn_decision_fusion.py`** - Neural Decision Fusion
 - **Status**: Production system
 - **Key Features**:
  - Multi-model prediction fusion
  - Neural network for weight learning
  - Dynamic model registration
 - **Good Ideas**:
  - ✅ Learnable model weights
  - ✅ Dynamic model registration
  - ✅ Neural fusion vs simple averaging
 ### 📊 Model Management Systems
 #### 1. **`model_manager.py`** - Comprehensive Model Manager
 - **Key Features**:
  - Model registry with metadata
  - Performance-based cleanup
  - Storage management
  - Model leaderboard
  - 2-action system migration support
 - **Good Ideas**:
  - ✅ Automated model lifecycle management
  - ✅ Performance-based retention
  - ✅ Storage monitoring
  - ✅ Model versioning
  - ✅ Metadata tracking
 #### 2. **`utils/checkpoint_manager.py`** - Checkpoint Management
 - **Good Ideas**:
  - ✅ Legacy model detection
  - ✅ Performance-based checkpoint saving
  - ✅ Metadata preservation
 ## Architectural Patterns & Good Ideas
 ### 🏗️ Architecture Patterns
 1. **Multi-Scale Processing**
   - Multiple kernel sizes (3,5,7,9,11,15)
   - Different attention scales
   - Temporal and spatial multi-scale
 2. **Attention Mechanisms**
   - Multi-head attention
   - Self-attention
   - Spatial attention
   - Cross-scale attention
   - Relative positional encoding
 3. **Residual Connections**
   - Pre-activation residual blocks
   - Enhanced residual connections
   - Memory barriers for gradient flow
 4. **Adaptive Architecture**
   - Dynamic network rebuilding
   - Input-shape aware models
   - Configurable model sizes
 5. **Normalization Strategies**
   - GroupNorm for batch_size=1
   - LayerNorm for transformers
   - BatchNorm for standard training
 ### 🔧 Training Innovations
 1. **Experience Replay Variants**
   - Priority experience replay
   - Example sifting datasets
   - Positive experience memory
 2. **Mixed Precision Training**
   - GPU optimization
   - Memory efficiency
   - Training speed improvements
 3. **Checkpoint Management**
   - Performance-based saving
   - Legacy model support
   - Metadata preservation
 4. **Model Fusion**
   - Neural decision fusion
   - Mixture of Experts
   - Dynamic weight learning
 ### 💡 Market-Specific Features
 1. **Order Book Integration**
   - COB-specific preprocessing
   - Market microstructure awareness
   - Imbalance calculations
 2. **Market Regime Detection**
   - Regime-aware models
   - Adaptive behavior
   - Context switching
 3. **Uncertainty Quantification**
   - Confidence estimation
   - Risk-aware decisions
   - Uncertainty propagation
 4. **Position Awareness**
   - Position-aware action selection
   - Risk management integration
   - Context-dependent decisions
 ## Recommendations for Cleanup
 ### ✅ Keep (Production Ready)
 - `NN/models/cnn_model.py` - Main production CNN
 - `NN/models/dqn_agent.py` - Main production DQN
 - `NN/models/cob_rl_model.py` - COB-specific RL
 - `core/nn_decision_fusion.py` - Decision fusion
 - `model_manager.py` - Model management
 - `utils/checkpoint_manager.py` - Checkpoint management
 ### 📦 Archive (Good Ideas, Not Currently Used)
 - `NN/models/advanced_transformer_trading.py` - Advanced transformer concepts
 - `NN/models/enhanced_cnn.py` - Alternative CNN architecture
 - `NN/models/transformer_model.py` - MoE and transformer concepts
 ### 🗑️ Remove (Redundant/Outdated)
 - `NN/models/cnn_model_pytorch.py` - Superseded by enhanced version
 - `NN/models/enhanced_cnn_with_orderbook.py` - Functionality integrated elsewhere
 - `NN/models/transformer_model_pytorch.py` - Basic implementation
 - `training/williams_market_structure.py` - Fallback no longer needed
 ### 🔄 Consolidate Ideas
 1. **Multi-scale attention** from advanced transformer → integrate into main CNN
 2. **Market regime detection** → integrate into orchestrator
 3. **Uncertainty estimation** → integrate into decision fusion
 4. **Relative positional encoding** → future transformer implementation
 5. **Experience replay variants** → integrate into main DQN
 ## Implementation Priority
 ### High Priority Integrations
 1. Multi-scale attention mechanisms
 2. Market regime detection
 3. Uncertainty quantification
 4. Enhanced experience replay
 ### Medium Priority
 1. Relative positional encoding
 2. Advanced normalization strategies
 3. Adaptive architecture features
 ### Low Priority
 1. MoE architecture
 2. Ultra-massive model variants
 3. TensorFlow migration features
 ## Conclusion
 The project contains many innovative ideas spread across multiple implementations. The cleanup should focus on:
 1. **Consolidating** the best features into production models
 2. **Archiving** implementations with unique concepts
 3. **Removing** redundant or superseded code
 4. **Documenting** architectural patterns for future reference
 The main production models (`cnn_model.py`, `dqn_agent.py`, `cob_rl_model.py`) should be enhanced with the best ideas from alternative implementations before cleanup. 
--- a/MODEL_STATISTICS_IMPLEMENTATION_SUMMARY.md
+++ b/MODEL_STATISTICS_IMPLEMENTATION_SUMMARY.md
@ -1,156 +0,0 @@
 # Model Statistics Implementation Summary
 ## Overview
 Successfully implemented comprehensive model statistics tracking for the TradingOrchestrator, providing real-time monitoring of model performance, inference rates, and loss tracking.
 ## Features Implemented
 ### 1. ModelStatistics Dataclass
 Created a comprehensive statistics tracking class with the following metrics:
 - **Inference Timing**: Last inference time, total inferences, inference rates (per second/minute)
 - **Loss Tracking**: Current loss, average loss, best/worst loss with rolling history
 - **Prediction History**: Last prediction, confidence, and rolling history of recent predictions
 - **Performance Metrics**: Accuracy tracking and model-specific metadata
 ### 2. Real-time Statistics Tracking
 - **Automatic Updates**: Statistics are updated automatically during each model inference
 - **Rolling Windows**: Uses deque with configurable limits for memory efficiency
 - **Rate Calculation**: Dynamic calculation of inference rates based on actual timing
 - **Error Handling**: Robust error handling to prevent statistics failures from affecting predictions
 ### 3. Integration Points
 #### Model Registration
 - Statistics are automatically initialized when models are registered
 - Cleanup happens automatically when models are unregistered
 - Each model gets its own dedicated statistics object
 #### Prediction Loop Integration
 - Statistics are updated in `_get_all_predictions` for each model inference
 - Tracks both successful predictions and failed inference attempts
 - Minimal performance overhead with efficient data structures
 #### Training Integration
 - Loss values are automatically tracked when models are trained
 - Updates both the existing `model_states` and new `model_statistics`
 - Provides historical loss tracking for trend analysis
 ### 4. Access Methods
 #### Individual Model Statistics
 ```python
 # Get statistics for a specific model
 stats = orchestrator.get_model_statistics("dqn_agent")
 print(f"Total inferences: {stats.total_inferences}")
 print(f"Inference rate: {stats.inference_rate_per_minute:.1f}/min")
 ```
 #### All Models Summary
 ```python
 # Get serializable summary of all models
 summary = orchestrator.get_model_statistics_summary()
 for model_name, stats in summary.items():
    print(f"{model_name}: {stats}")
 ```
 #### Logging and Monitoring
 ```python
 # Log current statistics (brief or detailed)
 orchestrator.log_model_statistics()  # Brief
 orchestrator.log_model_statistics(detailed=True)  # Detailed
 ```
 ## Test Results
 The implementation was successfully tested with the following results:
 ### Initial State
 - All models start with 0 inferences and no statistics
 - Statistics objects are properly initialized during model registration
 ### After 5 Prediction Batches
 - **dqn_agent**: 5 inferences, 63.5/min rate, last prediction: BUY (1.000 confidence)
 - **enhanced_cnn**: 5 inferences, 64.2/min rate, last prediction: SELL (0.499 confidence)  
 - **cob_rl_model**: 5 inferences, 65.3/min rate, last prediction: SELL (0.684 confidence)
 - **extrema_trainer**: 0 inferences (not being called in current setup)
 ### Key Observations
 1. **Accurate Rate Calculation**: Inference rates are calculated correctly based on actual timing
 2. **Proper Tracking**: Each model's predictions and confidence levels are tracked accurately
 3. **Memory Efficiency**: Rolling windows prevent unlimited memory growth
 4. **Error Resilience**: Statistics continue to work even when training fails
 ## Data Structure
 ### ModelStatistics Fields
 ```python
@dataclass
 class ModelStatistics:
    model_name: str
    last_inference_time: Optional[datetime] = None
    total_inferences: int = 0
    inference_rate_per_minute: float = 0.0
    inference_rate_per_second: float = 0.0
    current_loss: Optional[float] = None
    average_loss: Optional[float] = None
    best_loss: Optional[float] = None
    worst_loss: Optional[float] = None
    accuracy: Optional[float] = None
    last_prediction: Optional[str] = None
    last_confidence: Optional[float] = None
    inference_times: deque = field(default_factory=lambda: deque(maxlen=100))
    losses: deque = field(default_factory=lambda: deque(maxlen=100))
    predictions_history: deque = field(default_factory=lambda: deque(maxlen=50))
 ```
 ### JSON Serializable Summary
 The `get_model_statistics_summary()` method returns a clean, JSON-serializable dictionary perfect for:
 - Dashboard integration
 - API responses
 - Logging and monitoring systems
 - Performance analysis tools
 ## Performance Impact
 - **Minimal Overhead**: Statistics updates add negligible latency to predictions
 - **Memory Efficient**: Rolling windows prevent memory leaks
 - **Non-blocking**: Statistics failures don't affect model predictions
 - **Scalable**: Supports unlimited number of models
 ## Future Enhancements
 1. **Accuracy Calculation**: Implement prediction accuracy tracking based on market outcomes
 2. **Performance Alerts**: Add thresholds for inference rate drops or loss spikes
 3. **Historical Analysis**: Export statistics for long-term performance analysis
 4. **Dashboard Integration**: Real-time statistics display in trading dashboard
 5. **Model Comparison**: Comparative analysis tools for model performance
 ## Usage Examples
 ### Basic Monitoring
 ```python
 # Log current status
 orchestrator.log_model_statistics()
 # Get specific model performance
 dqn_stats = orchestrator.get_model_statistics("dqn_agent")
 if dqn_stats.inference_rate_per_minute < 10:
    logger.warning("DQN inference rate is low!")
 ```
 ### Dashboard Integration
 ```python
 # Get all statistics for dashboard
 stats_summary = orchestrator.get_model_statistics_summary()
 dashboard.update_model_metrics(stats_summary)
 ```
 ### Performance Analysis
 ```python
 # Analyze model performance trends
 for model_name, stats in orchestrator.model_statistics.items():
    recent_losses = list(stats.losses)
    if len(recent_losses) > 10:
        trend = "improving" if recent_losses[-1] < recent_losses[0] else "degrading"
        print(f"{model_name} loss trend: {trend}")
 ```
 This implementation provides comprehensive model monitoring capabilities while maintaining the system's performance and reliability.
--- a/ORCHESTRATOR_STREAMLINING_PLAN.md
+++ b/ORCHESTRATOR_STREAMLINING_PLAN.md
@ -1,229 +0,0 @@
 # Orchestrator Architecture Streamlining Plan
 ## Current State Analysis
 ### Basic TradingOrchestrator (`core/orchestrator.py`)
 - **Size**: 880 lines
 - **Purpose**: Core trading decisions, model coordination
 - **Features**:
  - Model registry and weight management
  - CNN and RL prediction combination
  - Decision callbacks
  - Performance tracking
  - Basic RL state building
 ### Enhanced TradingOrchestrator (`core/enhanced_orchestrator.py`)
 - **Size**: 5,743 lines (6.5x larger!)
 - **Inherits from**: TradingOrchestrator
 - **Additional Features**:
  - Universal Data Adapter (5 timeseries)
  - COB Integration
  - Neural Decision Fusion
  - Multi-timeframe analysis
  - Market regime detection
  - Sensitivity learning
  - Pivot point analysis
  - Extrema detection
  - Context data management
  - Williams market structure
  - Microstructure analysis
  - Order flow analysis
  - Cross-asset correlation
  - PnL-aware features
  - Trade flow features
  - Market impact estimation
  - Retrospective CNN training
  - Cold start predictions
 ## Problems Identified
 ### 1. **Massive Feature Bloat**
 - Enhanced orchestrator has become a "god object" with too many responsibilities
 - Single class doing: trading, analysis, training, data processing, market structure, etc.
 - Violates Single Responsibility Principle
 ### 2. **Code Duplication**
 - Many features reimplemented instead of extending base functionality
 - Similar RL state building in both classes
 - Overlapping market analysis
 ### 3. **Maintenance Nightmare**
 - 5,743 lines in single file is unmaintainable
 - Complex interdependencies
 - Hard to test individual components
 - Performance issues due to size
 ### 4. **Resource Inefficiency**
 - Loading entire enhanced orchestrator even if only basic features needed
 - Memory overhead from unused features
 - Slower initialization
 ## Proposed Solution: Modular Architecture
 ### 1. **Keep Streamlined Base Orchestrator**
 ```
 TradingOrchestrator (core/orchestrator.py)
 ├── Basic decision making
 ├── Model coordination
 ├── Performance tracking
 └── Core RL state building
 ```
 ### 2. **Create Modular Extensions**
 ```
 core/
 ├── orchestrator.py (Basic - 880 lines)
 ├── modules/
 │   ├── cob_module.py           # COB integration
 │   ├── market_analysis_module.py  # Market regime, volatility
 │   ├── multi_timeframe_module.py  # Multi-TF analysis
 │   ├── neural_fusion_module.py    # Neural decision fusion
 │   ├── pivot_analysis_module.py   # Williams/pivot points
 │   ├── extrema_module.py          # Extrema detection
 │   ├── microstructure_module.py   # Order flow analysis
 │   ├── correlation_module.py      # Cross-asset correlation
 │   └── training_module.py         # Advanced training features
 ```
 ### 3. **Configurable Enhanced Orchestrator**
 ```python
 class ConfigurableOrchestrator(TradingOrchestrator):
    def __init__(self, data_provider, modules=None):
        super().__init__(data_provider)
        self.modules = {}
        # Load only requested modules
        if modules:
            for module_name in modules:
                self.load_module(module_name)
    def load_module(self, module_name):
        # Dynamically load and initialize module
        pass
 ```
 ### 4. **Module Interface**
 ```python
 class OrchestratorModule:
    def __init__(self, orchestrator):
        self.orchestrator = orchestrator
    def initialize(self):
        pass
    def get_features(self, symbol):
        pass
    def get_predictions(self, symbol):
        pass
 ```
 ## Implementation Plan
 ### Phase 1: Extract Core Modules (Week 1)
 1. Extract COB integration to `cob_module.py`
 2. Extract market analysis to `market_analysis_module.py`
 3. Extract neural fusion to `neural_fusion_module.py`
 4. Test basic functionality
 ### Phase 2: Refactor Enhanced Features (Week 2)
 1. Move pivot analysis to `pivot_analysis_module.py`
 2. Move extrema detection to `extrema_module.py`
 3. Move microstructure analysis to `microstructure_module.py`
 4. Update imports and dependencies
 ### Phase 3: Create Configurable System (Week 3)
 1. Implement `ConfigurableOrchestrator`
 2. Create module loading system
 3. Add configuration file support
 4. Test different module combinations
 ### Phase 4: Clean Dashboard Integration (Week 4)
 1. Update dashboard to work with both Basic and Configurable
 2. Add module status display
 3. Dynamic feature enabling/disabling
 4. Performance optimization
 ## Benefits
 ### 1. **Maintainability**
 - Each module ~200-400 lines (manageable)
 - Clear separation of concerns
 - Individual module testing
 - Easier debugging
 ### 2. **Performance**
 - Load only needed features
 - Reduced memory footprint
 - Faster initialization
 - Better resource utilization
 ### 3. **Flexibility**
 - Mix and match features
 - Easy to add new modules
 - Configuration-driven setup
 - Development environment vs production
 ### 4. **Development**
 - Teams can work on individual modules
 - Clear interfaces reduce conflicts
 - Easier to add new features
 - Better code reuse
 ## Configuration Examples
 ### Minimal Setup (Basic Trading)
 ```yaml
 orchestrator:
  type: basic
  modules: []
 ```
 ### Full Enhanced Setup
 ```yaml
 orchestrator:
  type: configurable
  modules:
    - cob_module
    - neural_fusion_module
    - market_analysis_module
    - pivot_analysis_module
 ```
 ### Custom Setup (Research)
 ```yaml
 orchestrator:
  type: configurable
  modules:
    - market_analysis_module
    - extrema_module
    - training_module
 ```
 ## Migration Strategy
 ### 1. **Backward Compatibility**
 - Keep current Enhanced orchestrator as deprecated
 - Gradually migrate features to modules
 - Provide compatibility layer
 ### 2. **Gradual Migration**
 - Start with dashboard using Basic orchestrator
 - Add modules one by one
 - Test each integration
 ### 3. **Performance Testing**
 - Compare Basic vs Enhanced vs Modular
 - Memory usage analysis
 - Initialization time comparison
 - Decision-making speed tests
 ## Success Metrics
 1. **Code Size**: Enhanced orchestrator < 1,000 lines
 2. **Memory**: 50% reduction in memory usage for basic setup
 3. **Speed**: 3x faster initialization for basic setup
 4. **Maintainability**: Each module < 500 lines
 5. **Testing**: 90%+ test coverage per module
 This plan will transform the current monolithic enhanced orchestrator into a clean, modular, maintainable system while preserving all functionality and improving performance. 
--- a/PREDICTION_DATA_OPTIMIZATION_SUMMARY.md
+++ b/PREDICTION_DATA_OPTIMIZATION_SUMMARY.md
@ -1,96 +0,0 @@
 # Prediction Data Optimization Summary
 ## Problem Identified
 In the `_get_all_predictions` method, data was being fetched redundantly:
 1. **First fetch**: `_collect_model_input_data(symbol)` was called to get standardized input data
 2. **Second fetch**: Each individual prediction method (`_get_rl_prediction`, `_get_cnn_predictions`, `_get_generic_prediction`) called `build_base_data_input(symbol)` again
 3. **Third fetch**: Some methods like `_get_rl_state` also called `build_base_data_input(symbol)`
 This resulted in the same underlying data (technical indicators, COB data, OHLCV data) being fetched multiple times per prediction cycle.
 ## Solution Implemented
 ### 1. Centralized Data Fetching
 - Modified `_get_all_predictions` to fetch `BaseDataInput` once using `self.data_provider.build_base_data_input(symbol)`
 - Removed the redundant `_collect_model_input_data` method entirely
 ### 2. Updated Method Signatures
 All prediction methods now accept an optional `base_data` parameter:
 - `_get_rl_prediction(model, symbol, base_data=None)`
 - `_get_cnn_predictions(model, symbol, base_data=None)`
 - `_get_generic_prediction(model, symbol, base_data=None)`
 - `_get_rl_state(symbol, base_data=None)`
 ### 3. Backward Compatibility
 Each method maintains backward compatibility by building `BaseDataInput` if `base_data` is not provided, ensuring existing code continues to work.
 ### 4. Removed Redundant Code
 - Eliminated the `_collect_model_input_data` method (60+ lines of redundant code)
 - Removed duplicate `build_base_data_input` calls within prediction methods
 - Simplified the data flow architecture
 ## Benefits
 ### Performance Improvements
 - **Reduced API calls**: No more duplicate data fetching per prediction cycle
 - **Faster inference**: Single data fetch instead of 3-4 separate fetches
 - **Lower latency**: Predictions are generated faster due to reduced data overhead
 - **Memory efficiency**: Less temporary data structures created
 ### Code Quality
 - **DRY principle**: Eliminated code duplication
 - **Cleaner architecture**: Single source of truth for model input data
 - **Maintainability**: Easier to modify data fetching logic in one place
 - **Consistency**: All models now use the same data structure
 ### System Reliability
 - **Consistent data**: All models use exactly the same input data
 - **Reduced race conditions**: Single data fetch eliminates timing inconsistencies
 - **Error handling**: Centralized error handling for data fetching
 ## Technical Details
 ### Before Optimization
 ```python
 async def _get_all_predictions(self, symbol: str):
    # First data fetch
    input_data = await self._collect_model_input_data(symbol)
    for model in models:
        if isinstance(model, RLAgentInterface):
            # Second data fetch inside _get_rl_prediction
            rl_prediction = await self._get_rl_prediction(model, symbol)
        elif isinstance(model, CNNModelInterface):
            # Third data fetch inside _get_cnn_predictions
            cnn_predictions = await self._get_cnn_predictions(model, symbol)
 ```
 ### After Optimization
 ```python
 async def _get_all_predictions(self, symbol: str):
    # Single data fetch for all models
    base_data = self.data_provider.build_base_data_input(symbol)
    for model in models:
        if isinstance(model, RLAgentInterface):
            # Pass pre-built data, no additional fetch
            rl_prediction = await self._get_rl_prediction(model, symbol, base_data)
        elif isinstance(model, CNNModelInterface):
            # Pass pre-built data, no additional fetch
            cnn_predictions = await self._get_cnn_predictions(model, symbol, base_data)
 ```
 ## Testing Results
 - ✅ Orchestrator initializes successfully
 - ✅ All prediction methods work without errors
 - ✅ Generated 3 predictions in test run
 - ✅ No performance degradation observed
 - ✅ Backward compatibility maintained
 ## Future Considerations
 - Consider caching `BaseDataInput` objects for even better performance
 - Monitor memory usage to ensure the optimization doesn't increase memory footprint
 - Add metrics to measure the performance improvement quantitatively
 This optimization significantly improves the efficiency of the prediction system while maintaining full functionality and backward compatibility.
--- a/STREAMLINED_2_ACTION_SYSTEM_SUMMARY.md
+++ b/STREAMLINED_2_ACTION_SYSTEM_SUMMARY.md
@ -1,231 +0,0 @@
 # Streamlined 2-Action Trading System
 ## Overview
 The trading system has been simplified and streamlined to use only 2 actions (BUY/SELL) with intelligent position management, eliminating the complexity of HOLD signals and separate training modes.
 ## Key Simplifications
 ### 1. **2-Action System Only**
 - **Actions**: BUY and SELL only (no HOLD)
 - **Logic**: Until we have a signal, we naturally hold
 - **Position Intelligence**: Smart position management based on current state
 ### 2. **Simplified Training Pipeline**
 - **Removed**: Separate CNN, RL, and training modes
 - **Integrated**: All training happens within the web dashboard
 - **Flow**: Data → Indicators → CNN → RL → Orchestrator → Execution
 ### 3. **Streamlined Entry Points**
 - **Test Mode**: System validation and component testing
 - **Web Mode**: Live trading with integrated training pipeline
 - **Removed**: All standalone training modes
 ## Position Management Logic
 ### Current Position: FLAT (No Position)
 - **BUY Signal** → Enter LONG position
 - **SELL Signal** → Enter SHORT position
 ### Current Position: LONG
 - **BUY Signal** → Ignore (already long)
 - **SELL Signal** → Close LONG position
 - **Consecutive SELL** → Close LONG and enter SHORT
 ### Current Position: SHORT
 - **SELL Signal** → Ignore (already short)
 - **BUY Signal** → Close SHORT position
 - **Consecutive BUY** → Close SHORT and enter LONG
 ## Threshold System
 ### Entry Thresholds (Higher - More Certain)
 - **Default**: 0.75 confidence required
 - **Purpose**: Ensure high-quality entries
 - **Logic**: Only enter positions when very confident
 ### Exit Thresholds (Lower - Easier to Exit)
 - **Default**: 0.35 confidence required
 - **Purpose**: Quick exits to preserve capital
 - **Logic**: Exit quickly when confidence drops
 ## System Architecture
 ### Data Flow
 ```
 Live Market Data
      ↓
 Technical Indicators & Pivot Points
      ↓
 CNN Model Predictions
      ↓
 RL Agent Enhancement
      ↓
 Enhanced Orchestrator (2-Action Logic)
      ↓
 Trading Execution
 ```
 ### Core Components
 #### 1. **Enhanced Orchestrator**
 - 2-action decision making
 - Position tracking and management
 - Different thresholds for entry/exit
 - Consecutive signal detection
 #### 2. **Integrated Training**
 - CNN training on real market data
 - RL agent learning from live trading
 - No separate training sessions needed
 - Continuous improvement during live trading
 #### 3. **Position Intelligence**
 - Real-time position tracking
 - Smart transition logic
 - Consecutive signal handling
 - Risk management through thresholds
 ## Benefits of 2-Action System
 ### 1. **Simplicity**
 - Easier to understand and debug
 - Clearer decision logic
 - Reduced complexity in training
 ### 2. **Efficiency**
 - Faster training convergence
 - Less action space to explore
 - More focused learning
 ### 3. **Real-World Alignment**
 - Mimics actual trading decisions
 - Natural position management
 - Clear entry/exit logic
 ### 4. **Development Speed**
 - Faster iteration cycles
 - Easier testing and validation
 - Simplified codebase maintenance
 ## Model Updates
 ### CNN Models
 - Updated to 2-action output (BUY/SELL)
 - Simplified prediction logic
 - Better training convergence
 ### RL Agents
 - 2-action space for faster learning
 - Position-aware reward system
 - Integrated with live trading
 ## Configuration
 ### Entry Points
 ```bash
 # Test system components
 python main_clean.py --mode test
 # Run live trading with integrated training
 python main_clean.py --mode web --port 8051
 ```
 ### Key Settings
 ```yaml
 orchestrator:
  entry_threshold: 0.75    # Higher threshold for entries
  exit_threshold: 0.35     # Lower threshold for exits
  symbols: ['ETH/USDT']
  timeframes: ['1s', '1m', '1h', '4h']
 ```
 ## Dashboard Features
 ### Position Tracking
 - Real-time position status
 - Entry/exit history
 - Consecutive signal detection
 - Performance metrics
 ### Training Integration
 - Live CNN training
 - RL agent adaptation
 - Real-time learning metrics
 - Performance optimization
 ### Performance Metrics
 - 2-action system specific metrics
 - Position-based analytics
 - Entry/exit effectiveness
 - Threshold optimization
 ## Technical Implementation
 ### Position Tracking
 ```python
 current_positions = {
    'ETH/USDT': {
        'side': 'LONG',        # LONG, SHORT, or FLAT
        'entry_price': 3500.0,
        'timestamp': datetime.now()
    }
 }
 ```
 ### Signal History
 ```python
 last_signals = {
    'ETH/USDT': {
        'action': 'BUY',
        'confidence': 0.82,
        'timestamp': datetime.now()
    }
 }
 ```
 ### Decision Logic
 ```python
 def make_2_action_decision(symbol, predictions, market_state):
    # Get best prediction
    signal = get_best_signal(predictions)
    position = get_current_position(symbol)
    # Apply position-aware logic
    if position == 'FLAT':
        return enter_position(signal)
    elif position == 'LONG' and signal == 'SELL':
        return close_or_reverse_position(signal)
    elif position == 'SHORT' and signal == 'BUY':
        return close_or_reverse_position(signal)
    else:
        return None  # No action needed
 ```
 ## Future Enhancements
 ### 1. **Dynamic Thresholds**
 - Adaptive threshold adjustment
 - Market condition based thresholds
 - Performance-based optimization
 ### 2. **Advanced Position Management**
 - Partial position sizing
 - Risk-based position limits
 - Correlation-aware positioning
 ### 3. **Enhanced Training**
 - Multi-symbol coordination
 - Advanced reward systems
 - Real-time model updates
 ## Conclusion
 The streamlined 2-action system provides:
 - **Simplified Development**: Easier to code, test, and maintain
 - **Faster Training**: Convergence with fewer actions to learn
 - **Realistic Trading**: Mirrors actual trading decisions
 - **Integrated Pipeline**: Continuous learning during live trading
 - **Better Performance**: More focused and efficient trading logic
 This system is designed for rapid development cycles and easy adaptation to changing market conditions while maintaining high performance through intelligent position management. 
--- a/TENSOR_OPERATION_FIXES_REPORT.md
+++ b/TENSOR_OPERATION_FIXES_REPORT.md
@ -1,105 +0,0 @@
 # Tensor Operation Fixes Report
 *Generated: 2024-12-19*
 ## 🎯 Issue Summary
 The orchestrator was experiencing critical tensor operation errors that prevented model predictions:
 1. **Softmax Error**: `softmax() received an invalid combination of arguments - got (tuple, dim=int)`
 2. **View Error**: `view size is not compatible with input tensor's size and stride`
 3. **Unpacking Error**: `cannot unpack non-iterable NoneType object`
 ## 🔧 Fixes Applied
 ### 1. DQN Agent Softmax Fix (`NN/models/dqn_agent.py`)
 **Problem**: Q-values tensor had incorrect dimensions for softmax operation.
 **Solution**: Added dimension checking and reshaping before softmax:
 ```python
 # Before
 sell_confidence = torch.softmax(q_values, dim=1)[0, 0].item()
 # After  
 if q_values.dim() == 1:
    q_values = q_values.unsqueeze(0)
 sell_confidence = torch.softmax(q_values, dim=1)[0, 0].item()
 ```
 **Impact**: Prevents tensor dimension mismatch errors in confidence calculations.
 ### 2. CNN Model View Operations Fix (`NN/models/cnn_model.py`)
 **Problem**: `.view()` operations failed due to non-contiguous tensor memory layout.
 **Solution**: Replaced `.view()` with `.reshape()` for automatic contiguity handling:
 ```python
 # Before
 x = x.view(x.shape[0], -1, x.shape[-1])
 embedded = embedded.view(batch_size, seq_len, -1).transpose(1, 2).contiguous()
 # After
 x = x.reshape(x.shape[0], -1, x.shape[-1])
 embedded = embedded.reshape(batch_size, seq_len, -1).transpose(1, 2).contiguous()
 ```
 **Impact**: Eliminates tensor stride incompatibility errors during CNN forward pass.
 ### 3. Generic Prediction Unpacking Fix (`core/orchestrator.py`)
 **Problem**: Model prediction methods returned different formats, causing unpacking errors.
 **Solution**: Added robust return value handling:
 ```python
 # Before
 action_probs, confidence = model.predict(feature_matrix)
 # After
 prediction_result = model.predict(feature_matrix)
 if isinstance(prediction_result, tuple) and len(prediction_result) == 2:
    action_probs, confidence = prediction_result
 elif isinstance(prediction_result, dict):
    action_probs = prediction_result.get('probabilities', None)
    confidence = prediction_result.get('confidence', 0.7)
 else:
    action_probs = prediction_result
    confidence = 0.7
 ```
 **Impact**: Prevents unpacking errors when models return different formats.
 ## 📊 Technical Details
 ### Root Causes
 1. **Tensor Dimension Mismatch**: DQN models sometimes output 1D tensors when 2D expected
 2. **Memory Layout Issues**: `.view()` requires contiguous memory, `.reshape()` handles non-contiguous
 3. **API Inconsistency**: Different models return predictions in different formats
 ### Best Practices Applied
 - **Defensive Programming**: Check tensor dimensions before operations
 - **Memory Safety**: Use `.reshape()` instead of `.view()` for flexibility
 - **API Robustness**: Handle multiple return formats gracefully
 ## 🎯 Expected Results
 After these fixes:
 - ✅ DQN predictions should work without softmax errors
 - ✅ CNN predictions should work without view/stride errors  
 - ✅ Generic model predictions should work without unpacking errors
 - ✅ Orchestrator should generate proper trading decisions
 ## 🔄 Testing Recommendations
 1. **Run Dashboard**: Test that predictions are generated successfully
 2. **Monitor Logs**: Check for reduction in tensor operation errors
 3. **Verify Trading Signals**: Ensure BUY/SELL/HOLD decisions are made
 4. **Performance Check**: Confirm no significant performance degradation
 ## 📝 Notes
 - Some linter errors remain but are related to missing attributes, not tensor operations
 - The core tensor operation issues have been resolved
 - Models should now make predictions without crashing the orchestrator 
--- a/TRADING_ENHANCEMENTS_SUMMARY.md
+++ b/TRADING_ENHANCEMENTS_SUMMARY.md
@ -1,165 +0,0 @@
 # Trading System Enhancements Summary
 ## 🎯 **Issues Fixed**
 ### 1. **Position Sizing Issues**
 - **Problem**: Tiny position sizes (0.000 quantity) with meaningless P&L
 - **Solution**: Implemented percentage-based position sizing with leverage
 - **Result**: Meaningful position sizes based on account balance percentage
 ### 2. **Symbol Restrictions**
 - **Problem**: Both BTC and ETH trades were executing
 - **Solution**: Added `allowed_symbols: ["ETH/USDT"]` restriction
 - **Result**: Only ETH/USDT trades are now allowed
 ### 3. **Win Rate Calculation**
 - **Problem**: Incorrect win rate (50% instead of 69.2% for 9W/4L)
 - **Solution**: Fixed rounding issues in win/loss counting logic
 - **Result**: Accurate win rate calculations
 ### 4. **Missing Hold Time**
 - **Problem**: No way to debug model behavior timing
 - **Solution**: Added hold time tracking in seconds
 - **Result**: Each trade now shows exact hold duration
 ## 🚀 **New Features Implemented**
 ### 1. **Percentage-Based Position Sizing**
 ```yaml
 # config.yaml
 base_position_percent: 5.0     # 5% base position of account
 max_position_percent: 20.0     # 20% max position of account  
 min_position_percent: 2.0      # 2% min position of account
 leverage: 50.0                 # 50x leverage (adjustable in UI)
 simulation_account_usd: 100.0  # $100 simulation account
 ```
 **How it works:**
 - Base position = Account Balance × Base % × Confidence
 - Effective position = Base position × Leverage
 - Example: $100 account × 5% × 0.8 confidence × 50x = $200 effective position
 ### 2. **Hold Time Tracking**
 ```python
@dataclass
 class TradeRecord:
    # ... existing fields ...
    hold_time_seconds: float = 0.0  # NEW: Hold time in seconds
 ```
 **Benefits:**
 - Debug model behavior patterns
 - Identify optimal hold times
 - Analyze trade timing efficiency
 ### 3. **Enhanced Trading Statistics**
 ```python
 # Now includes:
 - Total fees paid
 - Hold time per trade
 - Percentage-based position info
 - Leverage settings
 ```
 ### 4. **UI-Adjustable Leverage**
 ```python
 def get_leverage(self) -> float:
    """Get current leverage setting"""
 def set_leverage(self, leverage: float) -> bool:
    """Set leverage (for UI control)"""
 def get_account_info(self) -> Dict[str, Any]:
    """Get account information for UI display"""
 ```
 ## 📊 **Dashboard Improvements**
 ### 1. **Enhanced Closed Trades Table**
 ```
 Time     | Side | Size  | Entry    | Exit     | Hold (s) | P&L    | Fees
 02:33:44 | LONG | 0.080 | $2588.33 | $2588.11 | 30       | $50.00 | $1.00
 ```
 ### 2. **Improved Trading Statistics**
 ```
 Win Rate: 60.0% (3W/2L) | Avg Win: $50.00 | Avg Loss: $25.00 | Total Fees: $5.00
 ```
 ## 🔧 **Configuration Changes**
 ### Before:
 ```yaml
 max_position_value_usd: 50.0   # Fixed USD amounts
 min_position_value_usd: 10.0
 leverage: 10.0
 ```
 ### After:
 ```yaml
 base_position_percent: 5.0     # Percentage of account
 max_position_percent: 20.0     # Scales with account size
 min_position_percent: 2.0
 leverage: 50.0                 # Higher leverage for significant P&L
 simulation_account_usd: 100.0  # Clear simulation balance
 allowed_symbols: ["ETH/USDT"]  # ETH-only trading
 ```
 ## 📈 **Expected Results**
 With these changes, you should now see:
 1. **Meaningful Position Sizes**: 
   - 2-20% of account balance
   - With 50x leverage = $100-$1000 effective positions
 2. **Significant P&L Values**:
   - Instead of $0.01 profits, expect $10-$100+ moves
   - Proportional to leverage and position size
 3. **Accurate Statistics**:
   - Correct win rate calculations
   - Hold time analysis capabilities
   - Total fees tracking
 4. **ETH-Only Trading**:
   - No more BTC trades
   - Focused on ETH/USDT pairs only
 5. **Better Debugging**:
   - Hold time shows model behavior patterns
   - Percentage-based sizing scales with account
   - UI-adjustable leverage for testing
 ## 🧪 **Test Results**
 All tests passing:
 - ✅ Position Sizing: Updated with percentage-based leverage
 - ✅ ETH-Only Trading: Configured in config
 - ✅ Win Rate Calculation: FIXED
 - ✅ New Features: WORKING
 ## 🎮 **UI Controls Available**
 The trading executor now supports:
 - `get_leverage()` - Get current leverage
 - `set_leverage(value)` - Adjust leverage from UI
 - `get_account_info()` - Get account status for display
 - Enhanced position and trade information
 ## 🔍 **Debugging Capabilities**
 With hold time tracking, you can now:
 - Identify if model holds positions too long/short
 - Correlate hold time with P&L success
 - Optimize entry/exit timing
 - Debug model behavior patterns
 Example analysis:
 ```
 Short holds (< 30s): 70% win rate
 Medium holds (30-60s): 60% win rate  
 Long holds (> 60s): 40% win rate
 ```
 This data helps optimize the model's decision timing! 
--- a/TRADING_FIXES_SUMMARY.md
+++ b/TRADING_FIXES_SUMMARY.md
@ -1,98 +0,0 @@
 # Trading System Fixes Summary
 ## Issues Identified
 After analyzing the trading data, we identified several critical issues in the trading system:
 1. **Duplicate Entry Prices**: The system was repeatedly entering trades at the same price ($3676.92 appeared in 9 out of 14 trades).
 2. **P&L Calculation Issues**: There were major discrepancies between the reported P&L and the expected P&L calculated from entry/exit prices and position size.
 3. **Trade Side Distribution**: All trades were SHORT positions, indicating a potential bias or configuration issue.
 4. **Rapid Consecutive Trades**: Several trades were executed within very short time frames (as low as 10-12 seconds apart).
 5. **Position Tracking Problems**: The system was not properly resetting position data between trades.
 ## Root Causes
 1. **Price Caching**: The `current_prices` dictionary was not being properly updated between trades, leading to stale prices being used for trade entries.
 2. **P&L Calculation Formula**: The P&L calculation was not correctly accounting for position side (LONG vs SHORT).
 3. **Missing Trade Cooldown**: There was no mechanism to prevent rapid consecutive trades.
 4. **Incomplete Position Cleanup**: When closing positions, the system was not fully cleaning up position data.
 5. **Dashboard Display Issues**: The dashboard was displaying incorrect P&L values due to calculation errors.
 ## Implemented Fixes
 ### 1. Price Caching Fix
 - Added a timestamp-based cache invalidation system
 - Force price refresh if cache is older than 5 seconds
 - Added logging for price updates
 ### 2. P&L Calculation Fix
 - Implemented correct P&L formula based on position side
 - For LONG positions: P&L = (exit_price - entry_price) * size
 - For SHORT positions: P&L = (entry_price - exit_price) * size
 - Added separate tracking for gross P&L, fees, and net P&L
 ### 3. Trade Cooldown System
 - Added a 30-second cooldown between trades for the same symbol
 - Prevents rapid consecutive entries that could lead to overtrading
 - Added blocking mechanism with reason tracking
 ### 4. Duplicate Entry Prevention
 - Added detection for entries at similar prices (within 0.1%)
 - Blocks trades that are too similar to recent entries
 - Added logging for blocked trades
 ### 5. Position Tracking Fix
 - Ensured complete position cleanup after closing
 - Added validation for position data
 - Improved position synchronization between executor and dashboard
 ### 6. Dashboard Display Fix
 - Fixed trade display to show accurate P&L values
 - Added validation for trade data
 - Improved error handling for invalid trades
 ## How to Apply the Fixes
 1. Run the `apply_trading_fixes.py` script to prepare the fix files:
   ```
   python apply_trading_fixes.py
   ```
 2. Run the `apply_trading_fixes_to_main.py` script to apply the fixes to the main.py file:
   ```
   python apply_trading_fixes_to_main.py
   ```
 3. Run the trading system with the fixes applied:
   ```
   python main.py
   ```
 ## Verification
 The fixes have been tested using the `test_trading_fixes.py` script, which verifies:
 - Price caching fix
 - Duplicate entry prevention
 - P&L calculation accuracy
 All tests pass, indicating that the fixes are working correctly.
 ## Additional Recommendations
 1. **Implement Bidirectional Trading**: The system currently shows a bias toward SHORT positions. Consider implementing balanced logic for both LONG and SHORT positions.
 2. **Add Trade Validation**: Implement additional validation for trade parameters (price, size, etc.) before execution.
 3. **Enhance Logging**: Add more detailed logging for trade execution and P&L calculation to help diagnose future issues.
 4. **Implement Circuit Breakers**: Add circuit breakers to halt trading if unusual patterns are detected (e.g., too many losing trades in a row).
 5. **Regular Audit**: Implement a regular audit process to check for trading anomalies and ensure P&L calculations are accurate.
--- a/TRAINING_SYSTEM_AUDIT_SUMMARY.md
+++ b/TRAINING_SYSTEM_AUDIT_SUMMARY.md
@ -1,185 +0,0 @@
 # Training System Audit and Fixes Summary
 ## Issues Identified and Fixed
 ### 1. **State Conversion Error in DQN Agent**
 **Problem**: DQN agent was receiving dictionary objects instead of numpy arrays, causing:
 ```
 Error validating state: float() argument must be a string or a real number, not 'dict'
 ```
 **Root Cause**: The training system was passing `BaseDataInput` objects or dictionaries directly to the DQN agent's `remember()` method, but the agent expected numpy arrays.
 **Solution**: Created a robust `_convert_to_rl_state()` method that handles multiple input formats:
 - `BaseDataInput` objects with `get_feature_vector()` method
 - Numpy arrays (pass-through)
 - Dictionaries with feature extraction
 - Lists/tuples with conversion
 - Single numeric values
 - Fallback to data provider
 ### 2. **Model Interface Training Method Access**
 **Problem**: Training methods existed in underlying models but weren't accessible through model interfaces.
 **Solution**: Modified training methods to access underlying models correctly:
 ```python
 # Get the underlying model from the interface
 underlying_model = getattr(model_interface, 'model', None)
 ```
 ### 3. **Model-Specific Training Logic**
 **Problem**: Generic training approach didn't account for different model architectures and training requirements.
 **Solution**: Implemented specialized training methods for each model type:
 - `_train_rl_model()` - For DQN agents with experience replay
 - `_train_cnn_model()` - For CNN models with training samples
 - `_train_cob_rl_model()` - For COB RL models with specific interfaces
 - `_train_generic_model()` - For other model types
 ### 4. **Data Type Validation and Sanitization**
 **Problem**: Models received inconsistent data types causing training failures.
 **Solution**: Added comprehensive data validation:
 - Ensure numpy array format
 - Convert object dtypes to float32
 - Handle non-finite values (NaN, inf)
 - Flatten multi-dimensional arrays when needed
 - Replace invalid values with safe defaults
 ## Implementation Details
 ### State Conversion Method
 ```python
 def _convert_to_rl_state(self, model_input, model_name: str) -> Optional[np.ndarray]:
    """Convert various model input formats to RL state numpy array"""
    # Method 1: BaseDataInput with get_feature_vector
    if hasattr(model_input, 'get_feature_vector'):
        state = model_input.get_feature_vector()
        if isinstance(state, np.ndarray):
            return state
    # Method 2: Already a numpy array
    if isinstance(model_input, np.ndarray):
        return model_input
    # Method 3: Dictionary with feature extraction
    # Method 4: List/tuple conversion
    # Method 5: Single numeric value
    # Method 6: Data provider fallback
 ```
 ### Enhanced RL Training
 ```python
 async def _train_rl_model(self, model, model_name: str, model_input, prediction: Dict, reward: float) -> bool:
    # Convert to proper state format
    state = self._convert_to_rl_state(model_input, model_name)
    # Validate state format
    if not isinstance(state, np.ndarray):
        return False
    # Handle object dtype conversion
    if state.dtype == object:
        state = state.astype(np.float32)
    # Sanitize data
    state = np.nan_to_num(state, nan=0.0, posinf=1.0, neginf=-1.0)
    # Add experience and train
    model.remember(state=state, action=action_idx, reward=reward, ...)
 ```
 ## Test Results
 ### State Conversion Tests
 ✅ **Test 1**: `numpy.ndarray` → `numpy.ndarray` (pass-through)
 ✅ **Test 2**: `dict` → `numpy.ndarray` (feature extraction)
 ✅ **Test 3**: `list` → `numpy.ndarray` (conversion)
 ✅ **Test 4**: `int` → `numpy.ndarray` (single value)
 ### Model Training Tests
 ✅ **DQN Agent**: Successfully adds experiences and triggers training
 ✅ **CNN Model**: Successfully adds training samples and trains in batches
 ✅ **COB RL Model**: Gracefully handles missing training methods
 ✅ **Generic Models**: Fallback methods work correctly
 ## Performance Improvements
 ### Before Fixes
 - ❌ Training failures due to data type mismatches
 - ❌ Dictionary objects passed to numeric functions
 - ❌ Inconsistent model interface access
 - ❌ Generic training approach for all models
 ### After Fixes
 - ✅ Robust data type conversion and validation
 - ✅ Proper numpy array handling throughout
 - ✅ Model-specific training logic
 - ✅ Graceful error handling and fallbacks
 - ✅ Comprehensive logging for debugging
 ## Error Handling Improvements
 ### Graceful Degradation
 - If state conversion fails, training is skipped with warning
 - If model doesn't support training, acknowledged without error
 - Invalid data is sanitized rather than causing crashes
 - Fallback methods ensure training continues
 ### Enhanced Logging
 - Debug logs for state conversion process
 - Training method availability logging
 - Success/failure status for each training attempt
 - Data type and shape validation logging
 ## Model-Specific Enhancements
 ### DQN Agent Training
 - Proper experience replay with validated states
 - Batch size checking before training
 - Loss tracking and statistics updates
 - Memory management for experience buffer
 ### CNN Model Training
 - Training sample accumulation
 - Batch training when sufficient samples
 - Integration with CNN adapter
 - Loss tracking from training results
 ### COB RL Model Training
 - Support for `train_step` method
 - Proper tensor conversion for PyTorch
 - Target creation for supervised learning
 - Fallback to experience-based training
 ## Future Considerations
 ### Monitoring and Metrics
 - Track training success rates per model
 - Monitor state conversion performance
 - Alert on repeated training failures
 - Performance metrics for different input types
 ### Optimization Opportunities
 - Cache converted states for repeated use
 - Batch training across multiple models
 - Asynchronous training to reduce latency
 - Memory-efficient state storage
 ### Extensibility
 - Easy addition of new model types
 - Pluggable training method registration
 - Configurable training parameters
 - Model-specific training schedules
 ## Summary
 The training system audit successfully identified and fixed critical issues that were preventing proper model training. The key improvements include:
 1. **Robust Data Handling**: Comprehensive input validation and conversion
 2. **Model-Specific Logic**: Tailored training approaches for different architectures
 3. **Error Resilience**: Graceful handling of edge cases and failures
 4. **Enhanced Monitoring**: Better logging and statistics tracking
 5. **Performance Optimization**: Efficient data processing and memory management
 The system now correctly trains all model types with proper data validation, comprehensive error handling, and detailed monitoring capabilities.
--- a/UNIVERSAL_MODEL_TOGGLE_SYSTEM_SUMMARY.md
+++ b/UNIVERSAL_MODEL_TOGGLE_SYSTEM_SUMMARY.md
@ -1,168 +0,0 @@
 # Universal Model Toggle System - Implementation Summary
 ## 🎯 Problem Solved
 The original dashboard had hardcoded model toggle callbacks for specific models (DQN, CNN, COB_RL, Decision_Fusion). This meant:
 - ❌ Adding new models required manual code changes
 - ❌ Each model needed separate hardcoded callbacks
 - ❌ No support for dynamic model registration
 - ❌ Maintenance nightmare when adding/removing models
 ## ✅ Solution Implemented
 Created a **Universal Model Toggle System** that works with any model dynamically:
 ### Key Features
 1. **Dynamic Model Discovery**
   - Automatically detects all models from orchestrator's model registry
   - Supports models with or without interfaces
   - Works with both registered models and toggle-only models
 2. **Universal Callback Generation**
   - Single generic callback handler for all models
   - Automatically creates inference and training toggles for each model
   - No hardcoded model names or callbacks
 3. **Robust State Management**
   - Toggle states persist across sessions
   - Automatic initialization for new models
   - Backward compatibility with existing models
 4. **Dynamic Model Registration**
   - Add new models at runtime without code changes
   - Remove models dynamically
   - Automatic callback creation for new models
 ## 🏗️ Architecture Changes
 ### 1. Dashboard (`web/clean_dashboard.py`)
 **Before:**
 ```python
 # Hardcoded model state variables
 self.dqn_inference_enabled = True
 self.cnn_inference_enabled = True
 # ... separate variables for each model
 # Hardcoded callbacks for each model
@self.app.callback(Output('dqn-inference-toggle', 'value'), ...)
 def update_dqn_inference_toggle(value): ...
@self.app.callback(Output('cnn-inference-toggle', 'value'), ...)
 def update_cnn_inference_toggle(value): ...
 # ... separate callback for each model
 ```
 **After:**
 ```python
 # Dynamic model state management
 self.model_toggle_states = {}  # Dynamic storage
 # Universal callback setup
 self._setup_universal_model_callbacks()
 def _setup_universal_model_callbacks(self):
    available_models = self._get_available_models()
    for model_name in available_models.keys():
        self._create_model_toggle_callbacks(model_name)
 def _create_model_toggle_callbacks(self, model_name):
    # Creates both inference and training callbacks dynamically
    @self.app.callback(...)
    def update_model_inference_toggle(value):
        return self._handle_model_toggle(model_name, 'inference', value)
 ```
 ### 2. Orchestrator (`core/orchestrator.py`)
 **Enhanced with:**
 - `register_model_dynamically()` - Add models at runtime
 - `get_all_registered_models()` - Get all available models
 - Automatic toggle state initialization for new models
 - Notification system for toggle changes
 ### 3. Model Registry (`models/__init__.py`)
 **Enhanced with:**
 - `unregister_model()` - Remove models dynamically
 - `get_memory_stats()` - Memory usage tracking
 - `cleanup_all_models()` - Cleanup functionality
 ## 🧪 Test Results
 The test script `test_universal_model_toggles.py` demonstrates:
 ✅ **Test 1: Model Discovery** - Found 9 existing models automatically
 ✅ **Test 2: Dynamic Registration** - Successfully added new model at runtime
 ✅ **Test 3: Toggle State Management** - Proper state retrieval for all models
 ✅ **Test 4: State Updates** - Toggle changes work correctly
 ✅ **Test 5: Interface-less Models** - Models without interfaces work
 ✅ **Test 6: Dashboard Integration** - Dashboard sees all 14 models dynamically
 ## 🚀 Usage Examples
 ### Adding a New Model Dynamically
 ```python
 # Through orchestrator
 success = orchestrator.register_model_dynamically("new_model", model_interface)
 # Through dashboard
 success = dashboard.add_model_dynamically("new_model", model_interface)
 ```
 ### Checking Model States
 ```python
 # Get all available models
 models = orchestrator.get_all_registered_models()
 # Get specific model toggle state
 state = orchestrator.get_model_toggle_state("any_model_name")
 # Returns: {"inference_enabled": True, "training_enabled": False}
 ```
 ### Updating Toggle States
 ```python
 # Enable/disable inference or training for any model
 orchestrator.set_model_toggle_state("any_model", inference_enabled=False)
 orchestrator.set_model_toggle_state("any_model", training_enabled=True)
 ```
 ## 🎯 Benefits Achieved
 1. **Scalability**: Add unlimited models without code changes
 2. **Maintainability**: Single universal handler instead of N hardcoded callbacks
 3. **Flexibility**: Works with any model type (DQN, CNN, Transformer, etc.)
 4. **Robustness**: Automatic state management and persistence
 5. **Future-Proof**: New model types automatically supported
 ## 🔧 Technical Implementation Details
 ### Model Discovery Process
 1. Check orchestrator's model registry for registered models
 2. Check orchestrator's toggle states for additional models
 3. Merge both sources to get complete model list
 4. Create callbacks for all discovered models
 ### Callback Generation
 - Uses Python closures to create unique callbacks for each model
 - Each model gets both inference and training toggle callbacks
 - Callbacks use generic handler with model name parameter
 ### State Persistence
 - Toggle states saved to `data/ui_state.json`
 - Automatic loading on startup
 - New models get default enabled state
 ## 🎉 Result
 The inf and trn checkboxes now work for **ALL models** - existing and future ones. The system automatically:
 - Discovers all available models
 - Creates appropriate toggle controls
 - Manages state persistence
 - Supports dynamic model addition/removal
 **No more hardcoded model callbacks needed!** 🚀
--- a/core/dashboard_cnn_integration.py
+++ b/core/dashboard_cnn_integration.py
@ -1,365 +0,0 @@
 """
 Dashboard CNN Integration
 This module integrates the EnhancedCNNAdapter with the dashboard system,
 providing real-time training, predictions, and performance metrics display.
 """
 import logging
 import time
 import threading
 from datetime import datetime, timedelta
 from typing import Dict, List, Optional, Any, Tuple
 from collections import deque
 import numpy as np
 from .enhanced_cnn_adapter import EnhancedCNNAdapter
 from .standardized_data_provider import StandardizedDataProvider
 from .data_models import BaseDataInput, ModelOutput, create_model_output
 logger = logging.getLogger(__name__)
 class DashboardCNNIntegration:
    """
    CNN integration for the dashboard system
    This class:
    1. Manages CNN model lifecycle in the dashboard
    2. Provides real-time training and inference
    3. Tracks performance metrics for dashboard display
    4. Handles model predictions for chart overlay
    """
    def __init__(self, data_provider: StandardizedDataProvider, symbols: List[str] = None):
        """
        Initialize the dashboard CNN integration
        Args:
            data_provider: Standardized data provider
            symbols: List of symbols to process
        """
        self.data_provider = data_provider
        self.symbols = symbols or ['ETH/USDT', 'BTC/USDT']
        # Initialize CNN adapter
        self.cnn_adapter = EnhancedCNNAdapter(checkpoint_dir="models/enhanced_cnn")
        # Load best checkpoint if available
        self.cnn_adapter.load_best_checkpoint()
        # Performance tracking
        self.performance_metrics = {
            'total_predictions': 0,
            'total_training_samples': 0,
            'last_training_time': None,
            'last_inference_time': None,
            'training_loss_history': deque(maxlen=100),
            'accuracy_history': deque(maxlen=100),
            'inference_times': deque(maxlen=100),
            'training_times': deque(maxlen=100),
            'predictions_per_second': 0.0,
            'training_per_second': 0.0,
            'model_status': 'FRESH',
            'confidence_history': deque(maxlen=100),
            'action_distribution': {'BUY': 0, 'SELL': 0, 'HOLD': 0}
        }
        # Prediction cache for dashboard display
        self.prediction_cache = {}
        self.prediction_history = {symbol: deque(maxlen=1000) for symbol in self.symbols}
        # Training control
        self.training_enabled = True
        self.inference_enabled = True
        self.training_lock = threading.Lock()
        # Real-time processing
        self.is_running = False
        self.processing_thread = None
        logger.info(f"DashboardCNNIntegration initialized for symbols: {self.symbols}")
    def start_real_time_processing(self):
        """Start real-time CNN processing"""
        if self.is_running:
            logger.warning("Real-time processing already running")
            return
        self.is_running = True
        self.processing_thread = threading.Thread(target=self._real_time_processing_loop, daemon=True)
        self.processing_thread.start()
        logger.info("Started real-time CNN processing")
    def stop_real_time_processing(self):
        """Stop real-time CNN processing"""
        self.is_running = False
        if self.processing_thread:
            self.processing_thread.join(timeout=5)
        logger.info("Stopped real-time CNN processing")
    def _real_time_processing_loop(self):
        """Main real-time processing loop"""
        last_prediction_time = {}
        prediction_interval = 1.0  # Make prediction every 1 second
        while self.is_running:
            try:
                current_time = time.time()
                for symbol in self.symbols:
                    # Check if it's time to make a prediction for this symbol
                    if (symbol not in last_prediction_time or 
                        current_time - last_prediction_time[symbol] >= prediction_interval):
                        # Make prediction if inference is enabled
                        if self.inference_enabled:
                            self._make_prediction(symbol)
                            last_prediction_time[symbol] = current_time
                # Update performance metrics
                self._update_performance_metrics()
                # Sleep briefly to prevent overwhelming the system
                time.sleep(0.1)
            except Exception as e:
                logger.error(f"Error in real-time processing loop: {e}")
                time.sleep(1)
    def _make_prediction(self, symbol: str):
        """Make a prediction for a symbol"""
        try:
            start_time = time.time()
            # Get standardized input data
            base_data = self.data_provider.get_base_data_input(symbol)
            if base_data is None:
                logger.debug(f"No base data available for {symbol}")
                return
            # Make prediction
            model_output = self.cnn_adapter.predict(base_data)
            # Record inference time
            inference_time = time.time() - start_time
            self.performance_metrics['inference_times'].append(inference_time)
            # Update performance metrics
            self.performance_metrics['total_predictions'] += 1
            self.performance_metrics['last_inference_time'] = datetime.now()
            self.performance_metrics['confidence_history'].append(model_output.confidence)
            # Update action distribution
            action = model_output.predictions['action']
            self.performance_metrics['action_distribution'][action] += 1
            # Cache prediction for dashboard
            self.prediction_cache[symbol] = model_output
            self.prediction_history[symbol].append(model_output)
            # Store model output in data provider
            self.data_provider.store_model_output(model_output)
            logger.debug(f"CNN prediction for {symbol}: {action} ({model_output.confidence:.3f})")
        except Exception as e:
            logger.error(f"Error making prediction for {symbol}: {e}")
    def add_training_sample(self, symbol: str, actual_action: str, reward: float):
        """Add a training sample and trigger training if enabled"""
        try:
            if not self.training_enabled:
                return
            # Get base data for the symbol
            base_data = self.data_provider.get_base_data_input(symbol)
            if base_data is None:
                logger.debug(f"No base data available for training sample: {symbol}")
                return
            # Add training sample
            self.cnn_adapter.add_training_sample(base_data, actual_action, reward)
            # Update metrics
            self.performance_metrics['total_training_samples'] += 1
            # Train model periodically (every 10 samples)
            if self.performance_metrics['total_training_samples'] % 10 == 0:
                self._train_model()
        except Exception as e:
            logger.error(f"Error adding training sample: {e}")
    def _train_model(self):
        """Train the CNN model"""
        try:
            with self.training_lock:
                start_time = time.time()
                # Train model
                metrics = self.cnn_adapter.train(epochs=1)
                # Record training time
                training_time = time.time() - start_time
                self.performance_metrics['training_times'].append(training_time)
                # Update performance metrics
                self.performance_metrics['last_training_time'] = datetime.now()
                if 'loss' in metrics:
                    self.performance_metrics['training_loss_history'].append(metrics['loss'])
                if 'accuracy' in metrics:
                    self.performance_metrics['accuracy_history'].append(metrics['accuracy'])
                # Update model status
                if metrics.get('accuracy', 0) > 0.5:
                    self.performance_metrics['model_status'] = 'TRAINED'
                else:
                    self.performance_metrics['model_status'] = 'TRAINING'
                logger.info(f"CNN training completed: loss={metrics.get('loss', 0):.4f}, accuracy={metrics.get('accuracy', 0):.4f}")
        except Exception as e:
            logger.error(f"Error training CNN model: {e}")
    def _update_performance_metrics(self):
        """Update performance metrics for dashboard display"""
        try:
            current_time = time.time()
            # Calculate predictions per second (last 60 seconds)
            recent_inferences = [t for t in self.performance_metrics['inference_times'] 
                               if current_time - t <= 60]
            self.performance_metrics['predictions_per_second'] = len(recent_inferences) / 60.0
            # Calculate training per second (last 60 seconds)
            recent_trainings = [t for t in self.performance_metrics['training_times'] 
                              if current_time - t <= 60]
            self.performance_metrics['training_per_second'] = len(recent_trainings) / 60.0
        except Exception as e:
            logger.error(f"Error updating performance metrics: {e}")
    def get_dashboard_metrics(self) -> Dict[str, Any]:
        """Get metrics for dashboard display"""
        try:
            # Calculate current loss
            current_loss = (self.performance_metrics['training_loss_history'][-1] 
                          if self.performance_metrics['training_loss_history'] else 0.0)
            # Calculate current accuracy
            current_accuracy = (self.performance_metrics['accuracy_history'][-1] 
                              if self.performance_metrics['accuracy_history'] else 0.0)
            # Calculate average confidence
            avg_confidence = (np.mean(list(self.performance_metrics['confidence_history'])) 
                            if self.performance_metrics['confidence_history'] else 0.0)
            # Get latest prediction
            latest_prediction = None
            latest_symbol = None
            for symbol, prediction in self.prediction_cache.items():
                if latest_prediction is None or prediction.timestamp > latest_prediction.timestamp:
                    latest_prediction = prediction
                    latest_symbol = symbol
            # Format timing information
            last_inference_str = "None"
            last_training_str = "None"
            if self.performance_metrics['last_inference_time']:
                last_inference_str = self.performance_metrics['last_inference_time'].strftime("%H:%M:%S")
            if self.performance_metrics['last_training_time']:
                last_training_str = self.performance_metrics['last_training_time'].strftime("%H:%M:%S")
            return {
                'model_name': 'CNN',
                'model_type': 'cnn',
                'parameters': '50.0M',
                'status': self.performance_metrics['model_status'],
                'current_loss': current_loss,
                'accuracy': current_accuracy,
                'confidence': avg_confidence,
                'total_predictions': self.performance_metrics['total_predictions'],
                'total_training_samples': self.performance_metrics['total_training_samples'],
                'predictions_per_second': self.performance_metrics['predictions_per_second'],
                'training_per_second': self.performance_metrics['training_per_second'],
                'last_inference': last_inference_str,
                'last_training': last_training_str,
                'latest_prediction': {
                    'action': latest_prediction.predictions['action'] if latest_prediction else 'HOLD',
                    'confidence': latest_prediction.confidence if latest_prediction else 0.0,
                    'symbol': latest_symbol or 'ETH/USDT',
                    'timestamp': latest_prediction.timestamp.strftime("%H:%M:%S") if latest_prediction else "None"
                },
                'action_distribution': self.performance_metrics['action_distribution'].copy(),
                'training_enabled': self.training_enabled,
                'inference_enabled': self.inference_enabled
            }
        except Exception as e:
            logger.error(f"Error getting dashboard metrics: {e}")
            return {
                'model_name': 'CNN',
                'model_type': 'cnn',
                'parameters': '50.0M',
                'status': 'ERROR',
                'current_loss': 0.0,
                'accuracy': 0.0,
                'confidence': 0.0,
                'error': str(e)
            }
    def get_predictions_for_chart(self, symbol: str, timeframe: str = '1s', limit: int = 100) -> List[Dict[str, Any]]:
        """Get predictions for chart overlay"""
        try:
            if symbol not in self.prediction_history:
                return []
            predictions = list(self.prediction_history[symbol])[-limit:]
            chart_data = []
            for prediction in predictions:
                chart_data.append({
                    'timestamp': prediction.timestamp,
                    'action': prediction.predictions['action'],
                    'confidence': prediction.confidence,
                    'buy_probability': prediction.predictions.get('buy_probability', 0.0),
                    'sell_probability': prediction.predictions.get('sell_probability', 0.0),
                    'hold_probability': prediction.predictions.get('hold_probability', 0.0)
                })
            return chart_data
        except Exception as e:
            logger.error(f"Error getting predictions for chart: {e}")
            return []
    def set_training_enabled(self, enabled: bool):
        """Enable or disable training"""
        self.training_enabled = enabled
        logger.info(f"CNN training {'enabled' if enabled else 'disabled'}")
    def set_inference_enabled(self, enabled: bool):
        """Enable or disable inference"""
        self.inference_enabled = enabled
        logger.info(f"CNN inference {'enabled' if enabled else 'disabled'}")
    def get_model_info(self) -> Dict[str, Any]:
        """Get model information for dashboard"""
        return {
            'name': 'Enhanced CNN',
            'version': '1.0',
            'parameters': '50.0M',
            'input_shape': self.cnn_adapter.model.input_shape if self.cnn_adapter.model else 'Unknown',
            'device': str(self.cnn_adapter.device),
            'checkpoint_dir': self.cnn_adapter.checkpoint_dir,
            'training_samples': len(self.cnn_adapter.training_data),
            'max_training_samples': self.cnn_adapter.max_training_samples
        }
--- a/core/enhanced_cnn_integration.py
+++ b/core/enhanced_cnn_integration.py
@ -1,403 +0,0 @@
 """
 Enhanced CNN Integration for Dashboard
 This module integrates the EnhancedCNNAdapter with the dashboard, providing real-time
 training and inference capabilities.
 """
 import logging
 import threading
 import time
 from datetime import datetime
 from typing import Dict, List, Optional, Any, Union
 import os
 from .enhanced_cnn_adapter import EnhancedCNNAdapter
 from .standardized_data_provider import StandardizedDataProvider
 from .data_models import BaseDataInput, ModelOutput, create_model_output
 logger = logging.getLogger(__name__)
 class EnhancedCNNIntegration:
    """
    Integration of EnhancedCNNAdapter with the dashboard
    This class:
    1. Manages the EnhancedCNNAdapter lifecycle
    2. Provides real-time training and inference
    3. Collects and reports performance metrics
    4. Integrates with the dashboard's model visualization
    """
    def __init__(self, data_provider: StandardizedDataProvider, checkpoint_dir: str = "models/enhanced_cnn"):
        """
        Initialize the EnhancedCNNIntegration
        Args:
            data_provider: StandardizedDataProvider instance
            checkpoint_dir: Directory to store checkpoints
        """
        self.data_provider = data_provider
        self.checkpoint_dir = checkpoint_dir
        self.model_name = "enhanced_cnn_v1"
        # Create checkpoint directory if it doesn't exist
        os.makedirs(checkpoint_dir, exist_ok=True)
        # Initialize CNN adapter
        self.cnn_adapter = EnhancedCNNAdapter(checkpoint_dir=checkpoint_dir)
        # Load best checkpoint if available
        self.cnn_adapter.load_best_checkpoint()
        # Performance tracking
        self.inference_times = []
        self.training_times = []
        self.total_inferences = 0
        self.total_training_runs = 0
        self.last_inference_time = None
        self.last_training_time = None
        self.inference_rate = 0.0
        self.training_rate = 0.0
        self.daily_inferences = 0
        self.daily_training_runs = 0
        # Training settings
        self.training_enabled = True
        self.inference_enabled = True
        self.training_frequency = 10  # Train every N inferences
        self.training_batch_size = 32
        self.training_epochs = 1
        # Latest prediction
        self.latest_prediction = None
        self.latest_prediction_time = None
        # Training metrics
        self.current_loss = 0.0
        self.initial_loss = None
        self.best_loss = None
        self.current_accuracy = 0.0
        self.improvement_percentage = 0.0
        # Training thread
        self.training_thread = None
        self.training_active = False
        self.stop_training = False
        logger.info(f"EnhancedCNNIntegration initialized with model: {self.model_name}")
    def start_continuous_training(self):
        """Start continuous training in a background thread"""
        if self.training_thread is not None and self.training_thread.is_alive():
            logger.info("Continuous training already running")
            return
        self.stop_training = False
        self.training_thread = threading.Thread(target=self._continuous_training_loop, daemon=True)
        self.training_thread.start()
        logger.info("Started continuous training thread")
    def stop_continuous_training(self):
        """Stop continuous training"""
        self.stop_training = True
        logger.info("Stopping continuous training thread")
    def _continuous_training_loop(self):
        """Continuous training loop"""
        try:
            self.training_active = True
            logger.info("Starting continuous training loop")
            while not self.stop_training:
                # Check if training is enabled
                if not self.training_enabled:
                    time.sleep(5)
                    continue
                # Check if we have enough training samples
                if len(self.cnn_adapter.training_data) < self.training_batch_size:
                    logger.debug(f"Not enough training samples: {len(self.cnn_adapter.training_data)}/{self.training_batch_size}")
                    time.sleep(5)
                    continue
                # Train model
                start_time = time.time()
                metrics = self.cnn_adapter.train(epochs=self.training_epochs)
                training_time = time.time() - start_time
                # Update metrics
                self.training_times.append(training_time)
                if len(self.training_times) > 100:
                    self.training_times.pop(0)
                self.total_training_runs += 1
                self.daily_training_runs += 1
                self.last_training_time = datetime.now()
                # Calculate training rate
                if self.training_times:
                    avg_training_time = sum(self.training_times) / len(self.training_times)
                    self.training_rate = 1.0 / avg_training_time if avg_training_time > 0 else 0.0
                # Update loss and accuracy
                self.current_loss = metrics.get('loss', 0.0)
                self.current_accuracy = metrics.get('accuracy', 0.0)
                # Update initial loss if not set
                if self.initial_loss is None:
                    self.initial_loss = self.current_loss
                # Update best loss
                if self.best_loss is None or self.current_loss < self.best_loss:
                    self.best_loss = self.current_loss
                # Calculate improvement percentage
                if self.initial_loss is not None and self.initial_loss > 0:
                    self.improvement_percentage = ((self.initial_loss - self.current_loss) / self.initial_loss) * 100
                logger.info(f"Training completed: loss={self.current_loss:.4f}, accuracy={self.current_accuracy:.4f}, samples={metrics.get('samples', 0)}")
                # Sleep before next training
                time.sleep(10)
        except Exception as e:
            logger.error(f"Error in continuous training loop: {e}")
        finally:
            self.training_active = False
    def predict(self, symbol: str) -> Optional[ModelOutput]:
        """
        Make a prediction using the EnhancedCNN model
        Args:
            symbol: Trading symbol
        Returns:
            ModelOutput: Standardized model output
        """
        try:
            # Check if inference is enabled
            if not self.inference_enabled:
                return None
            # Get standardized input data
            base_data = self.data_provider.get_base_data_input(symbol)
            if base_data is None:
                logger.warning(f"Failed to get base data input for {symbol}")
                return None
            # Make prediction
            start_time = time.time()
            model_output = self.cnn_adapter.predict(base_data)
            inference_time = time.time() - start_time
            # Update metrics
            self.inference_times.append(inference_time)
            if len(self.inference_times) > 100:
                self.inference_times.pop(0)
            self.total_inferences += 1
            self.daily_inferences += 1
            self.last_inference_time = datetime.now()
            # Calculate inference rate
            if self.inference_times:
                avg_inference_time = sum(self.inference_times) / len(self.inference_times)
                self.inference_rate = 1.0 / avg_inference_time if avg_inference_time > 0 else 0.0
            # Store latest prediction
            self.latest_prediction = model_output
            self.latest_prediction_time = datetime.now()
            # Store model output in data provider
            self.data_provider.store_model_output(model_output)
            # Add training sample if we have a price
            current_price = self._get_current_price(symbol)
            if current_price and current_price > 0:
                # Simulate market feedback based on price movement
                # In a real system, this would be replaced with actual market performance data
                action = model_output.predictions['action']
                # For demonstration, we'll use a simple heuristic:
                # - If price is above 3000, BUY is good
                # - If price is below 3000, SELL is good
                # - Otherwise, HOLD is good
                if current_price > 3000:
                    best_action = 'BUY'
                elif current_price < 3000:
                    best_action = 'SELL'
                else:
                    best_action = 'HOLD'
                # Calculate reward based on whether the action matched the best action
                if action == best_action:
                    reward = 0.05  # Positive reward for correct action
                else:
                    reward = -0.05  # Negative reward for incorrect action
                # Add training sample
                self.cnn_adapter.add_training_sample(base_data, best_action, reward)
                logger.debug(f"Added training sample for {symbol}, action: {action}, best_action: {best_action}, reward: {reward:.4f}")
            return model_output
        except Exception as e:
            logger.error(f"Error making prediction: {e}")
            return None
    def _get_current_price(self, symbol: str) -> Optional[float]:
        """Get current price for a symbol"""
        try:
            # Try to get price from data provider
            if hasattr(self.data_provider, 'current_prices'):
                binance_symbol = symbol.replace('/', '').upper()
                if binance_symbol in self.data_provider.current_prices:
                    return self.data_provider.current_prices[binance_symbol]
            # Try to get price from latest OHLCV data
            df = self.data_provider.get_historical_data(symbol, '1s', 1)
            if df is not None and not df.empty:
                return float(df.iloc[-1]['close'])
            return None
        except Exception as e:
            logger.error(f"Error getting current price: {e}")
            return None
    def get_model_state(self) -> Dict[str, Any]:
        """
        Get model state for dashboard display
        Returns:
            Dict[str, Any]: Model state
        """
        try:
            # Format prediction for display
            prediction_info = "FRESH"
            confidence = 0.0
            if self.latest_prediction:
                action = self.latest_prediction.predictions.get('action', 'UNKNOWN')
                confidence = self.latest_prediction.confidence
                # Map action to display text
                if action == 'BUY':
                    prediction_info = "BUY_SIGNAL"
                elif action == 'SELL':
                    prediction_info = "SELL_SIGNAL"
                elif action == 'HOLD':
                    prediction_info = "HOLD_SIGNAL"
                else:
                    prediction_info = "PATTERN_ANALYSIS"
            # Format timing information
            inference_timing = "None"
            training_timing = "None"
            if self.last_inference_time:
                inference_timing = self.last_inference_time.strftime('%H:%M:%S')
            if self.last_training_time:
                training_timing = self.last_training_time.strftime('%H:%M:%S')
            # Calculate improvement percentage
            improvement = 0.0
            if self.initial_loss is not None and self.initial_loss > 0 and self.current_loss > 0:
                improvement = ((self.initial_loss - self.current_loss) / self.initial_loss) * 100
            return {
                'model_name': self.model_name,
                'model_type': 'cnn',
                'parameters': 50000000,  # 50M parameters
                'status': 'ACTIVE' if self.inference_enabled else 'DISABLED',
                'checkpoint_loaded': True,  # Assume checkpoint is loaded
                'last_prediction': prediction_info,
                'confidence': confidence * 100,  # Convert to percentage
                'last_inference_time': inference_timing,
                'last_training_time': training_timing,
                'inference_rate': self.inference_rate,
                'training_rate': self.training_rate,
                'daily_inferences': self.daily_inferences,
                'daily_training_runs': self.daily_training_runs,
                'initial_loss': self.initial_loss,
                'current_loss': self.current_loss,
                'best_loss': self.best_loss,
                'current_accuracy': self.current_accuracy,
                'improvement_percentage': improvement,
                'training_active': self.training_active,
                'training_enabled': self.training_enabled,
                'inference_enabled': self.inference_enabled,
                'training_samples': len(self.cnn_adapter.training_data)
            }
        except Exception as e:
            logger.error(f"Error getting model state: {e}")
            return {
                'model_name': self.model_name,
                'model_type': 'cnn',
                'parameters': 50000000,  # 50M parameters
                'status': 'ERROR',
                'error': str(e)
            }
    def get_pivot_prediction(self) -> Dict[str, Any]:
        """
        Get pivot prediction for dashboard display
        Returns:
            Dict[str, Any]: Pivot prediction
        """
        try:
            if not self.latest_prediction:
                return {
                    'next_pivot': 0.0,
                    'pivot_type': 'UNKNOWN',
                    'confidence': 0.0,
                    'time_to_pivot': 0
                }
            # Extract pivot prediction from model output
            extrema_pred = self.latest_prediction.predictions.get('extrema', [0, 0, 0])
            # Determine pivot type (0=bottom, 1=top, 2=neither)
            pivot_type_idx = extrema_pred.index(max(extrema_pred))
            pivot_types = ['BOTTOM', 'TOP', 'RANGE_CONTINUATION']
            pivot_type = pivot_types[pivot_type_idx]
            # Get current price
            current_price = self._get_current_price('ETH/USDT') or 0.0
            # Calculate next pivot price (simple heuristic for demonstration)
            if pivot_type == 'BOTTOM':
                next_pivot = current_price * 0.95  # 5% below current price
            elif pivot_type == 'TOP':
                next_pivot = current_price * 1.05  # 5% above current price
            else:
                next_pivot = current_price  # Same as current price
            # Calculate confidence
            confidence = max(extrema_pred) * 100  # Convert to percentage
            # Calculate time to pivot (simple heuristic for demonstration)
            time_to_pivot = 5  # 5 minutes
            return {
                'next_pivot': next_pivot,
                'pivot_type': pivot_type,
                'confidence': confidence,
                'time_to_pivot': time_to_pivot
            }
        except Exception as e:
            logger.error(f"Error getting pivot prediction: {e}")
            return {
                'next_pivot': 0.0,
                'pivot_type': 'ERROR',
                'confidence': 0.0,
                'time_to_pivot': 0
            }
--- a/core/orchestrator.py
+++ b/core/orchestrator.py
@ -3369,12 +3369,17 @@ class TradingOrchestrator:
            )
            logger.info(f"  Outcome: {outcome_status}")
-            # Add performance summary
+            # Add comprehensive performance summary
            if model_name in self.model_performance:
                perf = self.model_performance[model_name]
                logger.info(
-                    f"  Performance: {perf['accuracy']:.1%} ({perf['correct']}/{perf['total']})"
+                    f"  Performance: {perf['directional_accuracy']:.1%} directional ({perf['directional_correct']}/{perf['total']}) | "
                    f"{perf['accuracy']:.1%} profitable ({perf['correct']}/{perf['total']})"
                )
                if perf["pivot_attempted"] > 0:
                    logger.info(
                        f"  Pivot Detection: {perf['pivot_accuracy']:.1%} ({perf['pivot_detected']}/{perf['pivot_attempted']})"
                    )
        except Exception as e:
            logger.error(f"Error in immediate training for {model_name}: {e}")
@ -3453,32 +3458,62 @@ class TradingOrchestrator:
                predicted_price_vector=predicted_price_vector,
            )
-            # Update model performance tracking
+            # Initialize enhanced model performance tracking
            if model_name not in self.model_performance:
                self.model_performance[model_name] = {
-                    "correct": 0,
+                    "correct": 0,  # Profitability accuracy (backwards compatible)
                    "total": 0,
-                    "accuracy": 0.0,
+                    "accuracy": 0.0,  # Profitability accuracy (backwards compatible)
                    "directional_correct": 0,  # NEW: Directional accuracy
                    "directional_accuracy": 0.0,  # NEW: Directional accuracy %
                    "pivot_detected": 0,  # NEW: Successful pivot detections
                    "pivot_attempted": 0,  # NEW: Total pivot attempts
                    "pivot_accuracy": 0.0,  # NEW: Pivot detection accuracy
                    "price_predictions": {"total": 0, "accurate": 0, "avg_error": 0.0},
                }
            # Ensure all new keys exist (for existing models)
            perf = self.model_performance[model_name]
            if "directional_correct" not in perf:
                perf["directional_correct"] = 0
                perf["directional_accuracy"] = 0.0
                perf["pivot_detected"] = 0
                perf["pivot_attempted"] = 0
                perf["pivot_accuracy"] = 0.0
            # Ensure price_predictions key exists
-            if "price_predictions" not in self.model_performance[model_name]:
+            if "price_predictions" not in perf:
-                self.model_performance[model_name]["price_predictions"] = {
+                perf["price_predictions"] = {"total": 0, "accurate": 0, "avg_error": 0.0}
                    "total": 0,
                    "accurate": 0,
                    "avg_error": 0.0,
                }
-            self.model_performance[model_name]["total"] += 1
+            # Calculate directional accuracy separately
-            if was_correct:
+            directional_correct = (
-                self.model_performance[model_name]["correct"] += 1
+                (predicted_action == "BUY" and price_change_pct > 0) or
-
+                (predicted_action == "SELL" and price_change_pct < 0) or
-            self.model_performance[model_name]["accuracy"] = (
+                (predicted_action == "HOLD" and abs(price_change_pct) < 0.05)
                self.model_performance[model_name]["correct"]
                / self.model_performance[model_name]["total"]
            )
            # Update all accuracy metrics
            perf["total"] += 1
            if was_correct:  # Profitability accuracy
                perf["correct"] += 1
            if directional_correct:
                perf["directional_correct"] += 1
            # Update pivot detection tracking
            is_significant_move = abs(price_change_pct) > 0.08  # 0.08% threshold for "significant"
            if predicted_action in ["BUY", "SELL"] and is_significant_move:
                perf["pivot_attempted"] += 1
                if directional_correct:
                    perf["pivot_detected"] += 1
            # Calculate all accuracy percentages
            perf["accuracy"] = perf["correct"] / perf["total"]  # Profitability accuracy
            perf["directional_accuracy"] = perf["directional_correct"] / perf["total"]  # Directional accuracy
            if perf["pivot_attempted"] > 0:
                perf["pivot_accuracy"] = perf["pivot_detected"] / perf["pivot_attempted"]  # Pivot accuracy
            else:
                perf["pivot_accuracy"] = 0.0
            # Track price prediction accuracy if available
            if inference_price is not None:
                price_prediction_stats = self.model_performance[model_name][
@ -3504,7 +3539,8 @@ class TradingOrchestrator:
                    f"({price_prediction_stats['avg_error']:.2f}% avg error)"
                )
-            # Enhanced logging for training evaluation
+            # Enhanced logging with new accuracy metrics
            perf = self.model_performance[model_name]
            logger.info(f"Training evaluation for {model_name}:")
            logger.info(
                f"  Action: {predicted_action} | Confidence: {prediction_confidence:.3f}"
@ -3512,10 +3548,15 @@ class TradingOrchestrator:
            logger.info(
                f"  Price change: {price_change_pct:+.3f}% | Time: {time_diff_seconds:.1f}s"
            )
-            logger.info(f"  Reward: {reward:.4f} | Correct: {was_correct}")
+            logger.info(f"  Reward: {reward:.4f} | Profitable: {was_correct} | Directional: {directional_correct}")
            logger.info(
-                f"  Accuracy: {self.model_performance[model_name]['accuracy']:.1%} ({self.model_performance[model_name]['correct']}/{self.model_performance[model_name]['total']})"
+                f"  Profitability: {perf['accuracy']:.1%} ({perf['correct']}/{perf['total']}) | "
                f"Directional: {perf['directional_accuracy']:.1%} ({perf['directional_correct']}/{perf['total']})"
            )
            if perf["pivot_attempted"] > 0:
                logger.info(
                    f"  Pivot Detection: {perf['pivot_accuracy']:.1%} ({perf['pivot_detected']}/{perf['pivot_attempted']})"
                )
            # Train the specific model based on sophisticated outcome
            await self._train_model_on_outcome(
@ -3549,6 +3590,45 @@ class TradingOrchestrator:
        except Exception as e:
            logger.error(f"Error evaluating and training on record: {e}")
    def _is_pivot_point(self, price_change_pct: float, prediction_confidence: float, time_diff_minutes: float) -> tuple[bool, str, float]:
        """
        Detect if this is a significant pivot point worth trading.
        Pivot points are the key moments where markets change direction or momentum.
        Returns:
            tuple: (is_pivot, pivot_type, pivot_strength)
        """
        abs_change = abs(price_change_pct)
        # Pivot point thresholds (much more realistic for crypto)
        minor_pivot = 0.08   # 0.08% - small but tradeable pivot
        medium_pivot = 0.25  # 0.25% - significant pivot
        major_pivot = 0.6    # 0.6% - major pivot
        massive_pivot = 1.2  # 1.2% - massive pivot
        # Time-based multipliers (faster pivots are more valuable)
        time_multiplier = 1.0
        if time_diff_minutes < 2.0:      # Very fast pivot
            time_multiplier = 2.0
        elif time_diff_minutes < 5.0:    # Fast pivot
            time_multiplier = 1.5
        elif time_diff_minutes > 15.0:   # Slow pivot - less valuable
            time_multiplier = 0.7
        # Confidence multiplier (high confidence pivots are more valuable)
        confidence_multiplier = 0.5 + (prediction_confidence * 1.5)  # 0.5 to 2.0
        if abs_change >= massive_pivot:
            return True, "MASSIVE_PIVOT", 10.0 * time_multiplier * confidence_multiplier
        elif abs_change >= major_pivot:
            return True, "MAJOR_PIVOT", 5.0 * time_multiplier * confidence_multiplier
        elif abs_change >= medium_pivot:
            return True, "MEDIUM_PIVOT", 2.5 * time_multiplier * confidence_multiplier
        elif abs_change >= minor_pivot:
            return True, "MINOR_PIVOT", 1.2 * time_multiplier * confidence_multiplier
        else:
            return False, "NO_PIVOT", 0.1  # Very small reward for noise
    def _calculate_sophisticated_reward(
        self,
        predicted_action: str,
@ -3562,10 +3642,18 @@ class TradingOrchestrator:
        predicted_price_vector: dict = None,
    ) -> tuple[float, bool]:
        """
-        Calculate sophisticated reward based on prediction accuracy, confidence, and price movement magnitude
+        PIVOT-POINT FOCUSED REWARD SYSTEM
-        Now considers position status and current P&L when evaluating decisions
+        
-        NOISE REDUCTION: Treats neutral/low-confidence signals as HOLD to reduce training noise
+        This system heavily rewards models for correctly identifying pivot points - 
-        PRICE VECTOR BONUS: Rewards accurate price direction and magnitude predictions
+        the actual profitable trading opportunities in the market. Small movements 
        are treated as noise and given minimal rewards.
        Key Features:
        - Separate directional accuracy vs profitability accuracy tracking
        - Heavy rewards for successful pivot point detection
        - Minimal penalties for noise (small movements)
        - Time-weighted rewards (faster detection = better)
        - Confidence-weighted rewards (higher confidence = better)
        Args:
            predicted_action: The predicted action ('BUY', 'SELL', 'HOLD')
@ -3579,21 +3667,36 @@ class TradingOrchestrator:
            predicted_price_vector: Dict with 'direction' (-1 to 1) and 'confidence' (0 to 1)
        Returns:
-            tuple: (reward, was_correct)
+            tuple: (reward, directional_correct, profitability_correct, pivot_detected)
        """
        try:
-            # NOISE REDUCTION: Treat low-confidence signals as HOLD
+            # Store original action for directional accuracy tracking
-            confidence_threshold = 0.6  # Only consider BUY/SELL if confidence > 60%
+            original_action = predicted_action
            if prediction_confidence < confidence_threshold:
                predicted_action = "HOLD"
                logger.debug(f"Low confidence ({prediction_confidence:.2f}) - treating as HOLD for noise reduction")
-            # FEE-AWARE THRESHOLDS: Account for trading fees (0.05-0.06% per trade, ~0.12% round trip)
+            # PIVOT POINT DETECTION
-            fee_cost = 0.12  # 0.12% round trip fee cost
+            is_pivot, pivot_type, pivot_strength = self._is_pivot_point(
-            movement_threshold = 0.15  # Minimum movement to be profitable after fees
+                price_change_pct, prediction_confidence, time_diff_minutes
-            strong_movement_threshold = 0.5  # Strong movements - good profit potential
+            )
-            rapid_movement_threshold = 1.0  # Rapid movements - excellent profit potential
+            
-            massive_movement_threshold = 2.0  # Massive movements - extraordinary profit potential
+            # DIRECTIONAL ACCURACY (simple direction prediction)
            directional_correct = False
            if predicted_action == "BUY" and price_change_pct > 0:
                directional_correct = True
            elif predicted_action == "SELL" and price_change_pct < 0:
                directional_correct = True
            elif predicted_action == "HOLD" and abs(price_change_pct) < 0.05:  # Very small movement
                directional_correct = True
            # PROFITABILITY ACCURACY (fee-aware profitable trades)
            fee_cost = 0.10  # 0.10% round trip fee cost (realistic for most exchanges)
            profitability_correct = False
            if predicted_action == "BUY" and price_change_pct > fee_cost:
                profitability_correct = True
            elif predicted_action == "SELL" and price_change_pct < -fee_cost:
                profitability_correct = True
            elif predicted_action == "HOLD" and abs(price_change_pct) < fee_cost:
                profitability_correct = True
            # Determine current position status if not provided
            if has_position is None and symbol:
@ -3604,210 +3707,104 @@ class TradingOrchestrator:
            elif has_position is None:
                has_position = False
-            # Determine if prediction was directionally correct
+            # PIVOT POINT REWARD CALCULATION
-            was_correct = False
+            base_reward = 0.0
-            directional_accuracy = 0.0
+            pivot_bonus = 0.0
-            if predicted_action == "BUY":
+            # For backwards compatibility, use profitability_correct as the main "was_correct"
-                # BUY signals need to overcome fee costs for profitability
+            was_correct = profitability_correct
                was_correct = price_change_pct > movement_threshold
-                # ENHANCED FEE-AWARE REWARD STRUCTURE
+            # MASSIVE REWARDS FOR SUCCESSFUL PIVOT POINT DETECTION
-                if price_change_pct > massive_movement_threshold:
+            if is_pivot and directional_correct:
-                    # Massive movements (2%+) - EXTRAORDINARY rewards for high confidence
+                # Base pivot reward
-                    directional_accuracy = price_change_pct * 5.0  # 5x multiplier for massive moves
+                base_reward = pivot_strength
                    if prediction_confidence > 0.8:
                        directional_accuracy *= 2.0  # Additional 2x for high confidence (10x total)
                elif price_change_pct > rapid_movement_threshold:
                    # Rapid movements (1%+) - EXCELLENT rewards for high confidence
                    directional_accuracy = price_change_pct * 3.0  # 3x multiplier for rapid moves
                    if prediction_confidence > 0.7:
                        directional_accuracy *= 1.5  # Additional 1.5x for good confidence (4.5x total)
                elif price_change_pct > strong_movement_threshold:
                    # Strong movements (0.5%+) - GOOD rewards
                    directional_accuracy = price_change_pct * 2.0  # 2x multiplier for strong moves
                else:
                    # Small movements - minimal rewards (fees eat most profit)
                    directional_accuracy = max(0, (price_change_pct - fee_cost)) * 0.5  # Penalty for fee cost
-            elif predicted_action == "SELL":
+                # EXTRAORDINARY bonuses for successful pivot predictions
-                # SELL signals need to overcome fee costs for profitability
+                if pivot_type == "MASSIVE_PIVOT":
-                was_correct = price_change_pct < -movement_threshold
+                    pivot_bonus = 50.0 * prediction_confidence  # Up to 50x reward!
                    logger.info(f"MASSIVE PIVOT SUCCESS: {pivot_type} detected with {prediction_confidence:.2f} confidence = {pivot_bonus:.1f}x bonus!")
                elif pivot_type == "MAJOR_PIVOT":
                    pivot_bonus = 20.0 * prediction_confidence  # Up to 20x reward!
                    logger.info(f"MAJOR PIVOT SUCCESS: {pivot_type} detected with {prediction_confidence:.2f} confidence = {pivot_bonus:.1f}x bonus!")
                elif pivot_type == "MEDIUM_PIVOT":
                    pivot_bonus = 8.0 * prediction_confidence   # Up to 8x reward!
                    logger.info(f"MEDIUM PIVOT SUCCESS: {pivot_type} detected with {prediction_confidence:.2f} confidence = {pivot_bonus:.1f}x bonus!")
                elif pivot_type == "MINOR_PIVOT":
                    pivot_bonus = 3.0 * prediction_confidence   # Up to 3x reward!
                    logger.info(f"MINOR PIVOT SUCCESS: {pivot_type} detected with {prediction_confidence:.2f} confidence = {pivot_bonus:.1f}x bonus!")
-                # ENHANCED FEE-AWARE REWARD STRUCTURE (symmetric to BUY)
+                # Additional time-based bonus for early detection
-                abs_change = abs(price_change_pct)
+                if time_diff_minutes < 1.0:
-                if abs_change > massive_movement_threshold:
+                    time_bonus = pivot_bonus * 0.5  # 50% bonus for very fast detection
-                    # Massive movements (2%+) - EXTRAORDINARY rewards for high confidence
+                    pivot_bonus += time_bonus
-                    directional_accuracy = abs_change * 5.0  # 5x multiplier for massive moves
+                    logger.info(f"EARLY DETECTION BONUS: Detected {pivot_type} in {time_diff_minutes:.1f}m = +{time_bonus:.1f} bonus")
                    if prediction_confidence > 0.8:
                        directional_accuracy *= 2.0  # Additional 2x for high confidence (10x total)
                elif abs_change > rapid_movement_threshold:
                    # Rapid movements (1%+) - EXCELLENT rewards for high confidence
                    directional_accuracy = abs_change * 3.0  # 3x multiplier for rapid moves
                    if prediction_confidence > 0.7:
                        directional_accuracy *= 1.5  # Additional 1.5x for good confidence (4.5x total)
                elif abs_change > strong_movement_threshold:
                    # Strong movements (0.5%+) - GOOD rewards
                    directional_accuracy = abs_change * 2.0  # 2x multiplier for strong moves
                else:
                    # Small movements - minimal rewards (fees eat most profit)
                    directional_accuracy = max(0, (abs_change - fee_cost)) * 0.5  # Penalty for fee cost
-            elif predicted_action == "HOLD":
+                base_reward += pivot_bonus
-                # HOLD evaluation with noise reduction - smaller rewards to reduce training noise
+                
-                if has_position:
+            elif is_pivot and not directional_correct:
-                    # If we have a position, HOLD evaluation depends on P&L and price movement
+                # MODERATE penalty for missing pivot points (still valuable to learn from)
-                    if current_position_pnl > 0:  # Currently profitable position
+                base_reward = -pivot_strength * 0.3  # Small penalty to encourage learning
-                        # Holding a profitable position is good if price continues favorably
+                logger.debug(f"MISSED PIVOT: {pivot_type} missed, small penalty = {base_reward:.2f}")
-                        if price_change_pct > 0:  # Price went up while holding profitable position - excellent
+                
-                            was_correct = True
+            elif not is_pivot and directional_correct:
-                            directional_accuracy = price_change_pct * 0.8  # Reduced from 1.5 to reduce noise
+                # Small reward for correct direction on non-pivots (noise)
-                        elif abs(price_change_pct) < movement_threshold:  # Price stable - good
+                base_reward = 0.2 * prediction_confidence
-                            was_correct = True
+                logger.debug(f"NOISE CORRECT: Correct direction on noise movement = {base_reward:.2f}")
                            directional_accuracy = movement_threshold * 0.5  # Reduced reward to reduce noise
                        else:  # Price dropped while holding profitable position - still okay but less reward
                            was_correct = True
                            directional_accuracy = max(0, (current_position_pnl / 100.0) - abs(price_change_pct) * 0.3)
                    elif current_position_pnl < 0:  # Currently losing position
                        # Holding a losing position is generally bad - should consider closing
                        if price_change_pct > movement_threshold:  # Price recovered - good hold
                            was_correct = True
                            directional_accuracy = price_change_pct * 0.6  # Reduced reward
                        else:  # Price continued down or stayed flat - bad hold
                            was_correct = False
                            # Penalty proportional to loss magnitude
                            directional_accuracy = abs(current_position_pnl / 100.0) * 0.3  # Reduced penalty
                    else:  # Breakeven position
                        # Standard HOLD evaluation for breakeven positions
                        if abs(price_change_pct) < movement_threshold:  # Price stable - good
                            was_correct = True
                            directional_accuracy = movement_threshold * 0.4  # Reduced reward
                        else:  # Price moved significantly - missed opportunity
                            was_correct = False
                            directional_accuracy = max(0, movement_threshold - abs(price_change_pct)) * 0.5
                else:
                    # If we don't have a position, HOLD is correct if price stayed relatively stable
                    was_correct = abs(price_change_pct) < movement_threshold
                    directional_accuracy = max(0, movement_threshold - abs(price_change_pct)) * 0.4  # Reduced reward
            # Calculate FEE-AWARE magnitude-based multiplier (aggressive rewards for profitable movements)
            abs_movement = abs(price_change_pct)
            if abs_movement > massive_movement_threshold:
                magnitude_multiplier = min(abs_movement / 1.0, 8.0)  # Up to 8x for massive moves (8% = 8x)
            elif abs_movement > rapid_movement_threshold:
                magnitude_multiplier = min(abs_movement / 1.5, 4.0)  # Up to 4x for rapid moves (6% = 4x)
            elif abs_movement > strong_movement_threshold:
                magnitude_multiplier = min(abs_movement / 2.0, 2.0)  # Up to 2x for strong moves (4% = 2x)
            else:
-                # Small movements get minimal multiplier due to fees
+                # Very small penalty for wrong direction on noise (don't overtrain on noise)
-                magnitude_multiplier = max(0.1, (abs_movement - fee_cost) / 2.0)  # Penalty for fee cost
+                base_reward = -0.1 * prediction_confidence
                logger.debug(f"NOISE INCORRECT: Wrong direction on noise movement = {base_reward:.2f}")
-            # Calculate confidence-based reward adjustment
+            # POSITION-AWARE ADJUSTMENTS
-            if was_correct:
+            if has_position:
-                # Reward confident correct predictions more, penalize unconfident correct predictions less
+                # Adjust rewards based on current position status
-                confidence_multiplier = 0.5 + (
+                if current_position_pnl > 0.5:  # Profitable position
-                    prediction_confidence * 1.5
+                    if predicted_action == "HOLD" and price_change_pct > 0:
-                )  # Range: 0.5 to 2.0
+                        base_reward += 0.5  # Bonus for holding profitable position during uptrend
-                base_reward = (
+                        logger.debug(f"POSITION BONUS: Holding profitable position during uptrend = +0.5")
-                    directional_accuracy * magnitude_multiplier * confidence_multiplier
+                elif current_position_pnl < -0.5:  # Losing position
                    if predicted_action in ["BUY", "SELL"] and directional_correct:
                        base_reward += 0.3  # Bonus for taking action to exit losing position
                        logger.debug(f"EXIT BONUS: Taking action on losing position = +0.3")
            # PRICE VECTOR BONUS (if available)
            if predicted_price_vector and isinstance(predicted_price_vector, dict):
                vector_bonus = self._calculate_price_vector_bonus(
                    predicted_price_vector, price_change_pct, abs(price_change_pct), prediction_confidence
                )
                if vector_bonus > 0:
                    base_reward += vector_bonus
                    logger.debug(f"PRICE VECTOR BONUS: +{vector_bonus:.3f}")
-                # ENHANCED HIGH-CONFIDENCE BONUSES for profitable movements
+            # Time decay factor (pivot detection should be fast)
-                abs_movement = abs(price_change_pct)
+            time_decay = max(0.3, 1.0 - (time_diff_minutes / 30.0))  # Decay over 30 minutes, min 30%
-                # Extraordinary confidence bonus for massive movements
+            # Apply time decay
                if prediction_confidence > 0.9 and abs_movement > massive_movement_threshold:
                    base_reward *= 3.0  # 300% bonus for ultra-confident massive moves
                    logger.info(f"ULTRA CONFIDENCE BONUS: {prediction_confidence:.2f} confidence + {abs_movement:.2f}% movement = 3x reward")
                # Excellent confidence bonus for rapid movements
                elif prediction_confidence > 0.8 and abs_movement > rapid_movement_threshold:
                    base_reward *= 2.0  # 200% bonus for very confident rapid moves
                    logger.info(f"HIGH CONFIDENCE BONUS: {prediction_confidence:.2f} confidence + {abs_movement:.2f}% movement = 2x reward")
                # Good confidence bonus for strong movements
                elif prediction_confidence > 0.7 and abs_movement > strong_movement_threshold:
                    base_reward *= 1.5  # 150% bonus for confident strong moves
                    logger.info(f"CONFIDENCE BONUS: {prediction_confidence:.2f} confidence + {abs_movement:.2f}% movement = 1.5x reward")
                # Rapid movement detection bonus (speed matters for fees)
                if time_diff_minutes < 5.0 and abs_movement > rapid_movement_threshold:
                    base_reward *= 1.3  # 30% bonus for rapid detection of big moves
                    logger.info(f"RAPID DETECTION BONUS: {abs_movement:.2f}% movement in {time_diff_minutes:.1f}m = 1.3x reward")
                # PRICE VECTOR ACCURACY BONUS - Reward models for accurate price direction/magnitude predictions
                if predicted_price_vector and isinstance(predicted_price_vector, dict):
                    vector_bonus = self._calculate_price_vector_bonus(
                        predicted_price_vector, price_change_pct, abs_movement, prediction_confidence
                    )
                    if vector_bonus > 0:
                        base_reward += vector_bonus
                        logger.info(f"PRICE VECTOR BONUS: +{vector_bonus:.3f} for accurate direction/magnitude prediction")
            else:
                # ENHANCED PENALTY SYSTEM: Discourage fee-losing trades
                abs_movement = abs(price_change_pct)
                # Penalize incorrect predictions more severely if they were confident
                confidence_penalty = 0.5 + (prediction_confidence * 1.5)  # Higher confidence = higher penalty
                base_penalty = abs_movement * confidence_penalty
                # SEVERE penalties for confident wrong predictions on big moves
                if prediction_confidence > 0.8 and abs_movement > rapid_movement_threshold:
                    base_penalty *= 5.0  # 5x penalty for very confident wrong on big moves
                    logger.warning(f"SEVERE PENALTY: {prediction_confidence:.2f} confidence wrong on {abs_movement:.2f}% movement = 5x penalty")
                elif prediction_confidence > 0.7 and abs_movement > strong_movement_threshold:
                    base_penalty *= 3.0  # 3x penalty for confident wrong on strong moves
                    logger.warning(f"HIGH PENALTY: {prediction_confidence:.2f} confidence wrong on {abs_movement:.2f}% movement = 3x penalty")
                elif prediction_confidence > 0.8:
                    base_penalty *= 2.0  # 2x penalty for overconfident wrong predictions
                # ADDITIONAL penalty for predictions that would lose money to fees
                if abs_movement < fee_cost and prediction_confidence > 0.5:
                    fee_loss_penalty = (fee_cost - abs_movement) * 2.0  # Penalty for fee-losing trades
                    base_penalty += fee_loss_penalty
                    logger.warning(f"FEE LOSS PENALTY: {abs_movement:.2f}% movement < {fee_cost:.2f}% fees = +{fee_loss_penalty:.3f} penalty")
                base_reward = -base_penalty
            # Time decay factor (predictions should be evaluated quickly)
            time_decay = max(
                0.1, 1.0 - (time_diff_minutes / 60.0)
            )  # Decay over 1 hour, min 10%
            # Final reward calculation
            final_reward = base_reward * time_decay
-            # Bonus for accurate price predictions
+            # Clamp reward to reasonable range (higher range for pivot bonuses)
-            if (
+            final_reward = max(-10.0, min(100.0, final_reward))
                has_price_prediction and abs(price_change_pct) < 1.0
            ):  # Accurate price prediction
                final_reward *= 1.2  # 20% bonus for accurate price predictions
                logger.debug(
                    f"Applied price prediction accuracy bonus: {final_reward:.3f}"
                )
-            # Clamp reward to reasonable range
+            # Log detailed accuracy information
-            final_reward = max(-5.0, min(5.0, final_reward))
+            logger.debug(
                f"REWARD CALCULATION: action={predicted_action}, confidence={prediction_confidence:.3f}, "
                f"price_change={price_change_pct:.3f}%, pivot={is_pivot}/{pivot_type}, "
                f"directional_correct={directional_correct}, profitability_correct={profitability_correct}, "
                f"reward={final_reward:.3f}"
            )
            return final_reward, was_correct
        except Exception as e:
            logger.error(f"Error calculating sophisticated reward: {e}")
-            # Fallback to simple reward with position awareness
+            # Fallback to simple directional accuracy
-            has_position = self._has_open_position(symbol) if symbol else False
+            simple_correct = (
-            
+                (predicted_action == "BUY" and price_change_pct > 0) or
-            if predicted_action == "HOLD" and has_position:
+                (predicted_action == "SELL" and price_change_pct < 0) or
-                # If holding a position, HOLD is correct if price didn't drop significantly
+                (predicted_action == "HOLD" and abs(price_change_pct) < 0.05)
-                simple_correct = price_change_pct > -0.2  # Allow small losses while holding
+            )
-            else:
+            return (1.0 if simple_correct else -0.1, simple_correct)
                # Standard evaluation for other cases
                simple_correct = (
                    (predicted_action == "BUY" and price_change_pct > 0.1)
                    or (predicted_action == "SELL" and price_change_pct < -0.1)
                    or (predicted_action == "HOLD" and abs(price_change_pct) < 0.1)
                )
            return (1.0 if simple_correct else -0.5, simple_correct)
    def _calculate_price_vector_bonus(
        self, 
@ -4334,6 +4331,25 @@ class TradingOrchestrator:
                # Create training sample from record
                model_input = record.get("model_input")
                # If model_input is None, try to generate fresh state for training
                if model_input is None:
                    logger.debug(f"No stored model input for {model_name}, generating fresh state")
                    try:
                        # Generate fresh input state for training
                        if hasattr(self, 'data_provider') and self.data_provider:
                            # Use data provider to generate current market state
                            fresh_state = self._generate_fresh_state_fallback(model_name)
                            if fresh_state is not None and len(fresh_state) > 0:
                                model_input = fresh_state
                                logger.debug(f"Generated fresh training state for {model_name}: shape={fresh_state.shape if hasattr(fresh_state, 'shape') else len(fresh_state)}")
                            else:
                                logger.warning(f"Failed to generate fresh state for {model_name}")
                        else:
                            logger.warning(f"No data provider available for generating fresh state for {model_name}")
                    except Exception as e:
                        logger.warning(f"Error generating fresh state for {model_name}: {e}")
                if model_input is not None:
                    # Convert to tensor and ensure device placement
                    device = next(self.cnn_model.parameters()).device
@ -4432,7 +4448,71 @@ class TradingOrchestrator:
                    )
                    return True
                else:
-                    logger.warning(f"No model input available for CNN training")
+                    logger.warning(f"No model input available for CNN training for {model_name}. This prevents the model from learning.")
                    # Try one more time to generate training data from current market conditions
                    try:
                        if hasattr(self, 'data_provider') and self.data_provider:
                            # Create minimal training sample from current market data
                            symbol = record.get("symbol", "ETH/USDT")
                            current_price = self._get_current_price(symbol)
                            # Get variables from function scope
                            actual_action = prediction["action"]
                            pred_confidence = prediction.get("confidence", 0.5)
                            # Create a basic feature vector (this is a fallback)
                            basic_features = np.array([
                                current_price / 10000.0,  # Normalized price
                                pred_confidence,           # Model confidence
                                reward,                    # Current reward
                                1.0 if actual_action == "BUY" else 0.0,
                                1.0 if actual_action == "SELL" else 0.0,
                                1.0 if actual_action == "HOLD" else 0.0
                            ], dtype=np.float32)
                            # Pad to expected size if needed
                            expected_size = 512  # Adjust based on your model's expected input size
                            if len(basic_features) < expected_size:
                                padding = np.zeros(expected_size - len(basic_features), dtype=np.float32)
                                basic_features = np.concatenate([basic_features, padding])
                            logger.info(f"Created fallback training features for {model_name}: shape={basic_features.shape}")
                            # Now perform training with the fallback features
                            device = next(self.cnn_model.parameters()).device
                            features_tensor = torch.tensor(basic_features, dtype=torch.float32, device=device).unsqueeze(0)
                            # Convert action to index
                            actions = ["BUY", "SELL", "HOLD"]
                            action_idx = actions.index(actual_action) if actual_action in actions else 2
                            action_tensor = torch.tensor([action_idx], dtype=torch.long, device=device)
                            reward_tensor = torch.tensor([reward], dtype=torch.float32, device=device)
                            # Perform minimal training step
                            self.cnn_model.train()
                            self.cnn_optimizer.zero_grad()
                            # Forward pass
                            q_values, _, _, _, _ = self.cnn_model(features_tensor)
                            # Calculate basic loss
                            q_values_selected = q_values.gather(1, action_tensor.unsqueeze(1)).squeeze(1)
                            loss = nn.MSELoss()(q_values_selected, reward_tensor)
                            # Backward pass
                            loss.backward()
                            torch.nn.utils.clip_grad_norm_(self.cnn_model.parameters(), max_norm=1.0)
                            self.cnn_optimizer.step()
                            logger.info(f"Fallback CNN training completed for {model_name}: loss={loss.item():.4f}")
                            return True
                    except Exception as fallback_error:
                        logger.error(f"Fallback CNN training failed for {model_name}: {fallback_error}")
                    # If we reach here, even fallback training failed
                    logger.error(f"All CNN training methods failed for {model_name}. Model will not learn from this prediction.")
                    return False
            # Try model interface training methods
--- a/data/ui_state.json
+++ b/data/ui_state.json
@ -14,16 +14,16 @@
        },
        "decision_fusion": {
            "inference_enabled": false,
-            "training_enabled": true
+            "training_enabled": false
        },
        "transformer": {
            "inference_enabled": false,
            "training_enabled": true
        },
        "dqn_agent": {
-            "inference_enabled": false,
+            "inference_enabled": true,
            "training_enabled": true
        }
    },
-    "timestamp": "2025-07-29T23:33:51.882579"
+    "timestamp": "2025-08-01T21:40:16.976016"
 }
--- a/mexc_captcha_tokens_20250703_022428.json
+++ b/mexc_captcha_tokens_20250703_022428.json
@ -1,12 +0,0 @@
 [
  {
    "token": "geetest eyJsb3ROdW1iZXIiOiI4NWFhM2Q3YjJkYmE0Mjk3YTQwODY0YmFhODZiMzA5NyIsImNhcHRjaGFPdXRwdXQiOiJaVkwzS3FWaWxnbEZjQWdXOENIQVgxMUVBLVVPUnE1aURQSldzcmlubDFqelBhRTNiUGlEc0VrVTJUR0xuUzRHV2k0N2JDa1hyREMwSktPWmwxX1dERkQwNWdSN1NkbFJ1Z2NDY0JmTGdLVlNBTEI0OUNrR200enZZcnZ3MUlkdnQ5RThRZURYQ2E0empLczdZMHByS3JEWV9SQW93S0d4OXltS0MxMlY0SHRzNFNYMUV1YnI1ZV9yUXZCcTZJZTZsNFVJMS1DTnc5RUhBaXRXOGU2TVZ6OFFqaGlUMndRM1F3eGxEWkpmZnF6M3VucUl5RTZXUnFSUEx1T0RQQUZkVlB3S3AzcWJTQ3JXcG5CTUFKOXFuXzV2UDlXNm1pR3FaRHZvSTY2cWRzcHlDWUMyWTV1RzJ0ZjZfRHRJaXhTTnhLWUU3cTlfcU1WR2ZJUzlHUXh6ZWg2Mkp2eG02SHZLdjFmXzJMa3FlcVkwRk94S2RxaVpyN2NkNjAxMHE5UlFJVDZLdmNZdU1Hcm04M2d4SnY1bXp4VkZCZWZFWXZfRjZGWFpnWXRMMmhWSDlQME42bHFXQkpCTUVicE1nRm0zbm1iZVBkaDYxeW12T0FUb2wyNlQ0Z2ZET2dFTVFhZTkxQlFNR2FVSFRSa2c3RGJIX2xMYXlBTHQ0TTdyYnpHSCIsInBhc3NUb2tlbiI6IjA0NmFkMGQ5ZjNiZGFmYzJhNDgwYzFiMjcyMmIzZDUzOTk5NTRmYWVlNTM1MTI1ZTQ1MjkzNzJjYWZjOGI5N2EiLCJnZW5UaW1lIjoiMTc1MTQ5ODY4NCJ9",
    "url": "https://www.mexc.com/ucgateway/captcha_api/captcha/robot/robot.future.openlong.ETH_USDT.300X",
    "timestamp": "2025-07-03T02:24:51.150716"
  },
  {
    "token": "geetest eyJsb3ROdW1iZXIiOiI5ZWVlMDQ2YTg1MmQ0MTU3YTNiYjdhM2M5MzJiNzJiYSIsImNhcHRjaGFPdXRwdXQiOiJaVkwzS3FWaWxnbEZjQWdXOENIQVgxMUVBLVVPUnE1aURQSldzcmlubDFqelBhRTNiUGlEc0VrVTJUR0xuUzRHZk9hVUhKRW1ZOS1FN0h3Q3NNV3hvbVZsNnIwZXRYZzIyWHBGdUVUdDdNS19Ud1J6NnotX2pCXzRkVDJqTnJRN0J3cExjQ25DNGZQUXQ5V040TWxrZ0NMU3p6MERNd09SeHJCZVRkVE5pSU5BdmdFRDZOMkU4a19XRmJ6SFZsYUtieElnM3dLSGVTMG9URU5DLUNaNElnMDJlS2x3UWFZY3liRnhKU2ZrWG1vekZNMDVJSHVDYUpwT0d2WXhhYS1YTWlDeGE0TnZlcVFqN2JwNk04Q09PSnNxNFlfa0pkX0Ruc2w0UW1memZCUTZseF9tenFCMnFweThxd3hKTFVYX0g3TGUyMXZ2bGtubG1KS0RSUEJtTWpUcGFiZ2F4M3Q1YzJmbHJhRjk2elhHQzVBdVVQY1FrbDIyOW0xSmlnMV83cXNfTjdpZFozd0hRcWZFZGxSYVRKQTR2U18yYnFlcGdLblJ3Y3oxaWtOOW1RaWNOSnpSNFNhdm1Pdi1BSzhwSEF0V2lkVjhrTkVYc3dGbUdSazFKQXBEX1hVUjlEdl9sNWJJNEFnbVJhcVlGdjhfRUNvN1g2cmt2UGZuOElTcCIsInBhc3NUb2tlbiI6IjRmZDFhZmU5NzI3MTk0ZGI3MDNlMDg2NWQ0ZDZjZTIyYzMwMzUyNzQ5NzVjMDIwNDFiNTY3Y2Y3MDdhYjM1OTMiLCJnZW5UaW1lIjoiMTc1MTQ5ODY5MiJ9",
    "url": "https://www.mexc.com/ucgateway/captcha_api/captcha/robot/robot.future.closelong.ETH_USDT.300X",
    "timestamp": "2025-07-03T02:24:57.885947"
  }
 ]
--- a/mexc_cookies_20250703_003625.json
+++ b/mexc_cookies_20250703_003625.json
@ -1,29 +0,0 @@
 {
  "bm_sv": "D92603BBC020E9C2CD11B2EBC8F22050~YAAQJKVf1NW5K7CXAQAAwtMVzRzHARcY60jrPVzy9G79fN3SY4z988SWHHxQlbPpyZHOj76c20AjCnS0QwveqzB08zcRoauoIe/sP3svlaIso9PIdWay0KIIVUe1XsiTJRfTm/DmS+QdrOuJb09rbfWLcEJF4/0QK7VY0UTzPTI2V3CMtxnmYjd1+tjfYsvt1R6O+Mw9mYjb7SjhRmiP/exY2UgZdLTJiqd+iWkc5Wejy5m6g5duOfRGtiA9mfs=~1",
  "bm_sz": "98D80FE4B23FE6352AE5194DA699FDDB~YAAQJKVf1GK4K7CXAQAAeQ0UzRw+aXiY5/Ujp+sZm0a4j+XAJFn6fKT4oph8YqIKF6uHSgXkFY3mBt8WWY98Y2w1QzOEFRkje8HTUYQgJsV59y5DIOTZKC6wutPD/bKdVi9ZKtk4CWbHIIRuCrnU1Nw2jqj5E0hsorhKGh8GeVsAeoao8FWovgdYD6u8Qpbr9aL5YZgVEIqJx6WmWLmcIg+wA8UFj8751Fl0B3/AGxY2pACUPjonPKNuX/UDYA5e98plOYUnYLyQMEGIapSrWKo1VXhKBDPLNedJ/Q2gOCGEGlj/u1Fs407QxxXwCvRSegL91y6modtL5JGoFucV1pYc4pgTwEAEdJfcLCEBaButTbaHI9T3SneqgCoGeatMMaqz0GHbvMD7fBQofARBqzN1L6aGlmmAISMzI3wx/SnsfXBl~3228228~3294529",
  "_abck": "0288E759712AF333A6EE15F66BC2A662~-1~YAAQJKVf1GC4K7CXAQAAeQ0UzQ77TfyX5SOWTgdW3DVqNFrTLz2fhLo2OC4I6ZHnW9qB0vwTjFDfOB65BwLSeFZoyVypVCGTtY/uL6f4zX0AxEGAU8tLg/jeO0acO4JpGrjYZSW1F56vEd9JbPU2HQPNERorgCDLQMSubMeLCfpqMp3VCW4w0Ssnk6Y4pBSs4mh0PH95v56XXDvat9k20/JPoK3Ip5kK2oKh5Vpk5rtNTVea66P0NBjVUw/EddRUuDDJpc8T4DtTLDXnD5SNDxEq8WDkrYd5kP4dNe0PtKcSOPYs2QLUbvAzfBuMvnhoSBaCjsqD15EZ3eDAoioli/LzsWSxaxetYfm0pA/s5HBXMdOEDi4V0E9b79N28rXcC8IJEHXtfdZdhJjwh1FW14lqF9iuOwER81wDEnIVtgwTwpd3ffrc35aNjb+kGiQ8W0FArFhUI/ZY2NDvPVngRjNrmRm0CsCm+6mdxxVNsGNMPKYG29mcGDi2P9HGDk45iOm0vzoaYUl1PlOh4VGq/V3QGbPYpkBsBtQUjrf/SQJe5IAbjCICTYlgxTo+/FAEjec+QdUsagTgV8YNycQfTK64A2bs1L1n+RO5tapLThU6NkxnUbqHOm6168RnT8ZRoAUpkJ5m3QpqSsuslnPRUPyxUr73v514jTBIUGsq4pUeRpXXd9FAh8Xkn4VZ9Bh3q4jP7eZ9Sv58mgnEVltNBFkeG3zsuIp5Hu69MSBU+8FD4gVlncbBinrTLNWRB8F00Gyvc03unrAznsTEyLiDq9guQf9tQNcGjxfggfnGq/Z1Gy/A7WMjiYw7pwGRVzAYnRgtcZoww9gQ/FdGkbp2Xl+oVZpaqFsHVvafWyOFr4pqQsmd353ddgKLjsEnpy/jcdUsIR/Ph3pYv++XlypXehXj0/GHL+WsosujJrYk4TuEsPKUcyHNr+r844mYUIhCYsI6XVKrq3fimdfdhmlkW8J1kZSTmFwP8QcwGlTK/mZDTJPyf8K5ugXcqOU8oIQzt5B2zfRwRYKHdhb8IUw=~-1~-1~-1",
  "RT": "\"z=1&dm=www.mexc.com&si=f5d53b58-7845-4db4-99f1-444e43d35199&ss=mcmh857q&sl=3&tt=90n&bcn=%2F%2F684dd311.akstat.io%2F&ld=1c9o\"",
  "mexc_fingerprint_visitorId": "tv1xchuZQbx9N0aBztUG",
  "_ga_L6XJCQTK75": "GS2.1.s1751492192$o1$g1$t1751492248$j4$l0$h0",
  "uc_token": "WEB66f893ede865e5d927efdea4a82e655ad5190239c247997d744ef9cd075f6f1e",
  "u_id": "WEB66f893ede865e5d927efdea4a82e655ad5190239c247997d744ef9cd075f6f1e",
  "_fbp": "fb.1.1751492193579.314807866777158389",
  "mxc_exchange_layout": "BA",
  "sensorsdata2015jssdkcross": "%7B%22distinct_id%22%3A%2221a8728990b84f4fa3ae64c8004b4aaa%22%2C%22first_id%22%3A%22197cd11dc751be-0dd66c04c69e96-26011f51-3686400-197cd11dc76189d%22%2C%22props%22%3A%7B%22%24latest_traffic_source_type%22%3A%22%E7%9B%B4%E6%8E%A5%E6%B5%81%E9%87%8F%22%2C%22%24latest_search_keyword%22%3A%22%E6%9C%AA%E5%8F%96%E5%88%B0%E5%80%BC_%E7%9B%B4%E6%8E%A5%E6%89%93%E5%BC%80%22%2C%22%24latest_referrer%22%3A%22%22%2C%22%24latest_landing_page%22%3A%22https%3A%2F%2Fwww.mexc.com%2Fen-GB%2Flogin%3Fprevious%3D%252Ffutures%252FETH_USDT%253Ftype%253Dlinear_swap%22%7D%2C%22identities%22%3A%22eyIkaWRlbnRpdHlfY29va2llX2lkIjoiMTk3Y2QxMWRjNzUxYmUtMGRkNjZjMDRjNjllOTYtMjYwMTFmNTEtMzY4NjQwMC0xOTdjZDExZGM3NjE4OWQiLCIkaWRlbnRpdHlfbG9naW5faWQiOiIyMWE4NzI4OTkwYjg0ZjRmYTNhZTY0YzgwMDRiNGFhYSJ9%22%2C%22history_login_id%22%3A%7B%22name%22%3A%22%24identity_login_id%22%2C%22value%22%3A%2221a8728990b84f4fa3ae64c8004b4aaa%22%7D%2C%22%24device_id%22%3A%22197cd11dc751be-0dd66c04c69e96-26011f51-3686400-197cd11dc76189d%22%7D",
  "mxc_theme_main": "dark",
  "mexc_fingerprint_requestId": "1751492199306.WMvKJd",
  "_ym_visorc": "b",
  "mexc_clearance_modal_show_date": "2025-07-03-undefined",
  "ak_bmsc": "35C21AA65F819E0BF9BEBDD10DCF7B70~000000000000000000000000000000~YAAQJKVf1BK2K7CXAQAAPAISzRwQdUOUs1H3HPAdl4COMFQAl+aEPzppLbdgrwA7wXbP/LZpxsYCFflUHDppYKUjzXyTZ9tIojSF3/6CW3OCiPhQo/qhf6XPbC4oQHpCNWaC9GJWEs/CGesQdfeBbhkXdfh+JpgmgCF788+x8IveDE9+9qaL/3QZRy+E7zlKjjvmMxBpahRy+ktY9/KMrCY2etyvtm91KUclr4k8HjkhtNJOlthWgUyiANXJtfbNUMgt+Hqgqa7QzSUfAEpxIXQ1CuROoY9LbU292LRN5TbtBy/uNv6qORT38rKsnpi7TGmyFSB9pj3YsoSzIuAUxYXSh4hXRgAoUQm3Yh5WdLp4ONeyZC1LIb8VCY5xXRy/VbfaHH1w7FodY1HpfHGKSiGHSNwqoiUmMPx13Rgjsgki4mE7bwFmG2H5WAilRIOZA5OkndEqGrOuiNTON7l6+g6mH0MzZ+/+3AjnfF2sXxFuV9itcs9x",
  "mxc_theme_upcolor": "upgreen",
  "_vid_t": "mQUFl49q1yLZhrL4tvOtFF38e+hGW5QoMS+eXKVD9Q4vQau6icnyipsdyGLW/FBukiO2ItK7EtzPIPMFrE5SbIeLSm1NKc/j+ZmobhX063QAlskf1x1J",
  "_ym_isad": "2",
  "_ym_d": "1751492196",
  "_ym_uid": "1751492196843266888",
  "bm_mi": "02862693F007017AEFD6639269A60D08~YAAQJKVf1Am2K7CXAQAAIf4RzRzNGqZ7Q3BC0kAAp/0sCOhHxxvEWTb7mBl8p7LUz0W6RZbw5Etz03Tvqu3H6+sb+yu1o0duU+bDflt7WLVSOfG5cA3im8Jeo6wZhqmxTu6gGXuBgxhrHw/RGCgcknxuZQiRM9cbM6LlZIAYiugFm2xzmO/1QcpjDhs4S8d880rv6TkMedlkYGwdgccAmvbaRVSmX9d5Yukm+hY+5GWuyKMeOjpatAhcgjShjpSDwYSpyQE7vVZLBp7TECIjI9uoWzR8A87YHScKYEuE08tb8YtGdG3O6g70NzasSX0JF3XTCjrVZA==~1",
  "_ga": "GA1.1.626437359.1751492192",
  "NEXT_LOCALE": "en-GB",
  "x-mxc-fingerprint": "tv1xchuZQbx9N0aBztUG",
  "CLIENT_LANG": "en-GB",
  "sajssdk_2015_cross_new_user": "1"
 }
--- a/mexc_cookies_20250703_010352.json
+++ b/mexc_cookies_20250703_010352.json
@ -1,28 +0,0 @@
 {
  "bm_sv": "5C10B638DC36B596422995FAFA8535C5~YAAQJKVf1MfUK7CXAQAA8NktzRwthLouCzg1Sqsm2yBQhAdvw8KbTCYRe0bzUrYEsQEahTebrBcYQoRF3+HyIAggj7MIsbFBANUqLcKJ66lD3QbuA3iU3MhUts/ZhA2dLaSoH5IbgdwiAd98s4bjsb3MSaNwI3nCEzWkLH2CZDyGJK6mhwHlA5VU6OXRLTVz+dfeh2n2fD0SbtcppFL2j9jqopWyKLaxQxYAg+Rs5g3xAo2BTa6/zmQ2YoxZR/w=~1",
  "bm_sz": "11FB853E475F9672ADEDFBC783F7487B~YAAQJKVf1G7UK7CXAQAAcY8tzRy3rXBghQVq4e094ZpjhvYRjSatbOxmR/iHhc0aV6NMJkhTwCOnCDsKjeU6sgcdpYgxkpgfhbvTgm5dQ7fEQ5cgmJtfNPmEisDQxZQIOXlI4yhgq7cks4jek9T9pxBx+iLtsZYy5LqIl7mqXc7R7MxMaWvDBfSVU1T0hY9DD0U3P4fxstSIVbGdRzcX2mvGNMcdTj3JMB1y9mXzKB44Prglw0zWa7BZT4imuh5OTQTY4OLNQM7gg5ERUHI7RTcxz+CAltGtBeMHTmWa+Jat/Cw9/DOP7Rud8fESZ7pmhmRE4Fe3Vp2/C+CW3qRnoptViXYOWr/sfKIKSlxIx+QF4Tw58tE5r2XbUVzAF0rQ2mLz9ASi5FnAgJi/DBRULeKhUMVPxsPhMWX5R25J3Gj5QnIED7PjttEt~3294770~3491121",
  "_abck": "F5684DE447CDB1B381EABA9AB94E79B7~-1~YAAQJKVf1GzUK7CXAQAAcY8tzQ60GFr2A1gYL72t6F06CTbh+67guEB40t7OXrDJpLYousPo1UKwE9/z804ie8unZxI7iZhwZO/AJfavIw2JHsMnYOhg8S8U/P+hTMOu0KvFYhMfmbSVSHEMInpzJlFPnFHcbYX1GtPn0US/FI8NeDxamlefbV4vHAYxQCWXp1RUVflOukD/ix7BGIvVqNdTQJDMfDY3UmNyu9JC88T8gFDUBxpTJvHNAzafWV7HTpSzLUmYzkFMp0Py39ZVOkVKgEwI9M15xseSNIzVBm6hm6DHwN9Z6ogDuaNsMkY3iJhL9+h75OTq2If9wNMiehwa5XeLHGfSYizXzUFJhuHdcEI1EZAowl2JKq4iGynNIom1/0v3focwlDFi93wxzpCXhCZBKnIRiIYGgS47zjS6kCZpYvuoBRnNvFx7tdJHMMkQQvx6+pk5UzmT4n3jUjS2WUTRoDuwiEvs5NDiO/Z2r4zHlpZnskDdpsDXT2SxvtMo1J451PCPSzt0merJ8vHZD5eLYE0tDBJaLMPzpW9MPHgW/OqrRc5QjcsdhHxNBnMGfhV2U0aHxVsuSuguZRPz7hGDRQJJXepAU8UzDM/d9KSYdMxUvSfcIk+48e3HHyodrKrfXh/0yIaeamsLeYE2na321B0DUoWe28DKbAIY3WdeYfH3WsGJ/LNrM43HeAe8Ng5Bw+5M0rO8m6MqGbaROvdt4JwBheY8g1jMcyXmXJWBAN0in+5F/sXph1sFdPxiiCc2uKQbyuBA34glvFz1JsbPGATEbicRvW0w88JlY3Ki8yNkEYxyFDv3n2C6R3I7Z/ZjdSJLVmS47sWnow1K6YAa31a3A8eVVFItran2v7S2QJBVmS7zb89yVO7oUq16z9a7o+0K5setv8d/jPkPIn9jgWcFOfVh7osl2g0vB/ZTmLoMvES5VxkWZPP3Uo9oIEyIaFzGq7ppYJ24SLj9I6wo9m5Xq9pup33F0Cpn2GyRzoxLpMm7bV/2EJ5eLBjJ3YFQRZxYf2NU1k2CJifFCfSQYOlhu7qCBxNWryWjQQgz9uvGqoKs~-1~-1~-1",
  "RT": "\"z=1&dm=www.mexc.com&si=5943fd2a-6403-43d4-87aa-b4ac4403c94f&ss=mcmi7gg2&sl=3&tt=6d5&bcn=%2F%2F02179916.akstat.io%2F&ld=2fhr\"",
  "mexc_fingerprint_visitorId": "tv1xchuZQbx9N0aBztUG",
  "_ga_L6XJCQTK75": "GS2.1.s1751493837$o1$g1$t1751493945$j59$l0$h0",
  "uc_token": "WEB3756d4bd507f4dc9e5c6732b16d40aa668a2e3aea55107801a42f40389c39b9c",
  "u_id": "WEB3756d4bd507f4dc9e5c6732b16d40aa668a2e3aea55107801a42f40389c39b9c",
  "_fbp": "fb.1.1751493843684.307329583674408195",
  "mxc_exchange_layout": "BA",
  "sensorsdata2015jssdkcross": "%7B%22distinct_id%22%3A%2221a8728990b84f4fa3ae64c8004b4aaa%22%2C%22first_id%22%3A%22197cd2b02f56f6-08b72b0d8e14ee-26011f51-3686400-197cd2b02f6b59%22%2C%22props%22%3A%7B%22%24latest_traffic_source_type%22%3A%22%E7%9B%B4%E6%8E%A5%E6%B5%81%E9%87%8F%22%2C%22%24latest_search_keyword%22%3A%22%E6%9C%AA%E5%8F%96%E5%88%B0%E5%80%BC_%E7%9B%B4%E6%8E%A5%E6%89%93%E5%BC%80%22%2C%22%24latest_referrer%22%3A%22%22%2C%22%24latest_landing_page%22%3A%22https%3A%2F%2Fwww.mexc.com%2Fen-GB%2Flogin%3Fprevious%3D%252Ffutures%252FETH_USDT%253Ftype%253Dlinear_swap%22%7D%2C%22identities%22%3A%22eyIkaWRlbnRpdHlfY29va2llX2lkIjoiMTk3Y2QyYjAyZjU2ZjYtMDhiNzJiMGQ4ZTE0ZWUtMjYwMTFmNTEtMzY4NjQwMC0xOTdjZDJiMDJmNmI1OSIsIiRpZGVudGl0eV9sb2dpbl9pZCI6IjIxYTg3Mjg5OTBiODRmNGZhM2FlNjRjODAwNGI0YWFhIn0%3D%22%2C%22history_login_id%22%3A%7B%22name%22%3A%22%24identity_login_id%22%2C%22value%22%3A%2221a8728990b84f4fa3ae64c8004b4aaa%22%7D%2C%22%24device_id%22%3A%22197cd2b02f56f6-08b72b0d8e14ee-26011f51-3686400-197cd2b02f6b59%22%7D",
  "mxc_theme_main": "dark",
  "mexc_fingerprint_requestId": "1751493848491.aXJWxX",
  "ak_bmsc": "10B7B90E8C6CA0B2242A59C6BE9D5D09~000000000000000000000000000000~YAAQJKVf1BnQK7CXAQAAJwsrzRyGc8OCIHU9sjkSsoX2E9ZroYaoxZCEToLh8uS5k28z0rzxl4Oi8eXg1oKxdWZslNQCj4/PExgD4O1++Wfi2KNovx4cUehcmbtiR3a28w+gNaiVpWAUPjPnUTaHLAr7cgVU/IOdoOC0cdvxaHThWtwIbVu+YsGazlnHiND1w3u7V0Yc1irC6ZONXqD2rIIZlntEOFiJGPTs8egY3xMLeSpI0tZYp8CASAKzxp/v96ugcPBMehwZ03ue6s6bi8qGYgF1IuOgVTFW9lPVzxCYjvH+ASlmppbLm/vrCUSPjtzJcTz/ySfvtMYaai8cv3CwCf/Ke51plRXJo0wIzGOpBzzJG5/GMA924kx1EQiBTgJptG0i7ZrgrfhqtBjjB2sU0ZBofFqmVu/VXLV6iOCQBHFtpZeI60oFARGoZFP2mYbfxeIKG8ERrQ==",
  "mexc_clearance_modal_show_date": "2025-07-03-undefined",
  "_ym_isad": "2",
  "_vid_t": "hRsGoNygvD+rX1A4eY/XZLO5cGWlpbA3XIXKtYTjDPFdunb5ACYp5eKitX9KQSQj/YXpG2PcnbPZDIpAVQ0AGjaUpR058ahvxYptRHKSGwPghgfLZQ==",
  "_ym_visorc": "b",
  "_ym_d": "1751493846",
  "_ym_uid": "1751493846425437427",
  "mxc_theme_upcolor": "upgreen",
  "NEXT_LOCALE": "en-GB",
  "x-mxc-fingerprint": "tv1xchuZQbx9N0aBztUG",
  "CLIENT_LANG": "en-GB",
  "_ga": "GA1.1.1034661072.1751493838",
  "sajssdk_2015_cross_new_user": "1"
 }
--- a/mexc_requests_20250703_003625.json
+++ b/mexc_requests_20250703_003625.json
--- a/mexc_requests_20250703_010352.json
+++ b/mexc_requests_20250703_010352.json
--- a/mexc_requests_20250703_015321.json
+++ b/mexc_requests_20250703_015321.json
--- a/mexc_requests_20250703_021049.json
+++ b/mexc_requests_20250703_021049.json
--- a/mexc_requests_20250703_022428.json
+++ b/mexc_requests_20250703_022428.json
--- a/mexc_requests_20250703_023536.json
+++ b/mexc_requests_20250703_023536.json
--- a/mexc_requests_20250703_024032.json
+++ b/mexc_requests_20250703_024032.json
--- a/test_queue_logging.py
+++ b/test_queue_logging.py
@ -1,173 +0,0 @@
 #!/usr/bin/env python3
 """
 Test Queue Logging
 Test the improved logging for FIFO queue status
 """
 from datetime import datetime
 from core.orchestrator import TradingOrchestrator
 from core.data_provider import DataProvider
 from core.data_models import OHLCVBar
 def test_insufficient_data_logging():
    """Test logging when there's insufficient data"""
    print("=== Testing Insufficient Data Logging ===")
    try:
        # Create orchestrator
        data_provider = DataProvider()
        orchestrator = TradingOrchestrator(data_provider)
        # Log initial empty queue status
        print("\n1. Initial queue status:")
        orchestrator.log_queue_status(detailed=True)
        # Try to build BaseDataInput with no data (should show detailed warnings)
        print("\n2. Attempting to build BaseDataInput with no data:")
        base_data = orchestrator.build_base_data_input('ETH/USDT')
        print(f"Result: {base_data is not None}")
        # Add some data but not enough
        print("\n3. Adding insufficient data (50 bars, need 100):")
        for i in range(50):
            test_bar = OHLCVBar(
                symbol="ETH/USDT",
                timestamp=datetime.now(),
                open=2500.0 + i,
                high=2510.0 + i,
                low=2490.0 + i,
                close=2505.0 + i,
                volume=1000.0 + i,
                timeframe="1s"
            )
            orchestrator.update_data_queue('ohlcv_1s', 'ETH/USDT', test_bar)
        # Log queue status after adding some data
        print("\n4. Queue status after adding 50 bars:")
        orchestrator.log_queue_status(detailed=True)
        # Try to build BaseDataInput again (should show we have 50, need 100)
        print("\n5. Attempting to build BaseDataInput with insufficient data:")
        base_data = orchestrator.build_base_data_input('ETH/USDT')
        print(f"Result: {base_data is not None}")
        # Add enough data for ohlcv_1s but not other timeframes
        print("\n6. Adding enough 1s data (150 total) but missing other timeframes:")
        for i in range(50, 150):
            test_bar = OHLCVBar(
                symbol="ETH/USDT",
                timestamp=datetime.now(),
                open=2500.0 + i,
                high=2510.0 + i,
                low=2490.0 + i,
                close=2505.0 + i,
                volume=1000.0 + i,
                timeframe="1s"
            )
            orchestrator.update_data_queue('ohlcv_1s', 'ETH/USDT', test_bar)
        # Try again (should show 1s is OK but 1m/1h/1d are missing)
        print("\n7. Attempting to build BaseDataInput with mixed data availability:")
        base_data = orchestrator.build_base_data_input('ETH/USDT')
        print(f"Result: {base_data is not None}")
        return True
    except Exception as e:
        print(f"❌ Test failed: {e}")
        return False
 def test_queue_status_logging():
    """Test detailed queue status logging"""
    print("\n=== Testing Queue Status Logging ===")
    try:
        data_provider = DataProvider()
        orchestrator = TradingOrchestrator(data_provider)
        # Add various types of data
        print("\n1. Adding mixed data types:")
        # Add OHLCV data
        for i in range(75):
            test_bar = OHLCVBar(
                symbol="ETH/USDT",
                timestamp=datetime.now(),
                open=2500.0 + i,
                high=2510.0 + i,
                low=2490.0 + i,
                close=2505.0 + i,
                volume=1000.0 + i,
                timeframe="1s"
            )
            orchestrator.update_data_queue('ohlcv_1s', 'ETH/USDT', test_bar)
        # Add some 1m data
        for i in range(25):
            test_bar = OHLCVBar(
                symbol="ETH/USDT",
                timestamp=datetime.now(),
                open=2500.0 + i,
                high=2510.0 + i,
                low=2490.0 + i,
                close=2505.0 + i,
                volume=1000.0 + i,
                timeframe="1m"
            )
            orchestrator.update_data_queue('ohlcv_1m', 'ETH/USDT', test_bar)
        # Add technical indicators
        indicators = {'rsi': 45.5, 'macd': 0.15, 'bb_upper': 2520.0}
        orchestrator.update_data_queue('technical_indicators', 'ETH/USDT', indicators)
        # Add BTC data
        for i in range(60):
            btc_bar = OHLCVBar(
                symbol="BTC/USDT",
                timestamp=datetime.now(),
                open=50000.0 + i,
                high=50100.0 + i,
                low=49900.0 + i,
                close=50050.0 + i,
                volume=100.0 + i,
                timeframe="1s"
            )
            orchestrator.update_data_queue('ohlcv_1s', 'BTC/USDT', btc_bar)
        print("\n2. Detailed queue status:")
        orchestrator.log_queue_status(detailed=True)
        print("\n3. Simple queue status:")
        orchestrator.log_queue_status(detailed=False)
        print("\n4. Attempting to build BaseDataInput:")
        base_data = orchestrator.build_base_data_input('ETH/USDT')
        print(f"Result: {base_data is not None}")
        return True
    except Exception as e:
        print(f"❌ Test failed: {e}")
        return False
 def main():
    """Run logging tests"""
    print("=== Queue Logging Test Suite ===")
    test1_passed = test_insufficient_data_logging()
    test2_passed = test_queue_status_logging()
    print(f"\n=== Results ===")
    print(f"Insufficient data logging: {'✅ PASSED' if test1_passed else '❌ FAILED'}")
    print(f"Queue status logging: {'✅ PASSED' if test2_passed else '❌ FAILED'}")
    if test1_passed and test2_passed:
        print("\n✅ ALL TESTS PASSED!")
        print("✅ Improved logging shows actual vs required data counts")
        print("✅ Detailed queue status provides debugging information")
    else:
        print("\n❌ Some tests failed")
 if __name__ == "__main__":
    main()
--- a/test_safe_logging.py
+++ b/test_safe_logging.py
@ -1,69 +0,0 @@
 #!/usr/bin/env python3
 """
 Test script to verify safe_logging functionality
 This script tests that the safe logging module properly handles:
 1. Non-ASCII characters (emojis, smart quotes)
 2. UTF-8 encoding
 3. Error handling on Windows
 """
 import logging
 from safe_logging import setup_safe_logging
 def test_safe_logging():
    """Test the safe logging module with various character types"""
    # Setup safe logging
    setup_safe_logging()
    logger = logging.getLogger(__name__)
    print("Testing Safe Logging Module...")
    print("=" * 50)
    # Test regular ASCII messages
    logger.info("Regular ASCII message - this should work fine")
    # Test messages with emojis
    logger.info("Testing emojis: 🚀 💰 📈 📊 🔥")
    # Test messages with smart quotes and special characters
    logger.info("Testing smart quotes: Hello World Test")
    # Test messages with various Unicode characters
    logger.info("Testing Unicode: café résumé naïve Ω α β γ δ")
    # Test messages with mixed content
    logger.info("Mixed content: Regular text with emojis 🎉 and quotes like this")
    # Test error messages with special characters
    logger.error("Error with special chars: ❌ Failed to process €100.50")
    # Test warning messages
    logger.warning("Warning with symbols: ⚠️ Temperature is 37°C")
    # Test debug messages
    logger.debug("Debug info: Processing file data.txt at 95% completion ✓")
    # Test exception handling with special characters
    try:
        raise ValueError("Error with emoji: 💥 Something went wrong!")
    except Exception as e:
        logger.exception("Exception caught with special chars: %s", str(e))
    # Test formatting with special characters
    symbol = "ETH/USDT"
    price = 2500.50
    change = 2.3
    logger.info(f"Price update for {symbol}: ${price:.2f} (+{change}% 📈)")
    # Test large message with many special characters
    large_msg = "Large message: " + "🔄" * 50 + " Processing complete ✅"
    logger.info(large_msg)
    print("=" * 50)
    print("✅ Safe logging test completed!")
    print("If you see this message, all logging calls were successful.")
    print("Check the log file at logs/safe_logging.log for the complete output.")
 if __name__ == "__main__":
    test_safe_logging()
--- a/test_signal_preservation.py
+++ b/test_signal_preservation.py
@ -1,99 +0,0 @@
 #!/usr/bin/env python3
 import time
 from web.clean_dashboard import CleanTradingDashboard
 from core.data_provider import DataProvider
 from core.orchestrator import TradingOrchestrator
 from core.trading_executor import TradingExecutor
 print('Testing signal preservation improvements...')
 # Create dashboard instance
 data_provider = DataProvider()
 orchestrator = TradingOrchestrator(data_provider)
 trading_executor = TradingExecutor()
 dashboard = CleanTradingDashboard(
    data_provider=data_provider,
    orchestrator=orchestrator,
    trading_executor=trading_executor
 )
 print(f'Initial recent_decisions count: {len(dashboard.recent_decisions)}')
 # Add test signals similar to the user's example
 test_signals = [
    {'timestamp': '20:39:32', 'action': 'HOLD', 'confidence': 0.01, 'price': 2420.07},
    {'timestamp': '20:39:02', 'action': 'HOLD', 'confidence': 0.01, 'price': 2416.89},
    {'timestamp': '20:38:45', 'action': 'BUY', 'confidence': 0.65, 'price': 2415.23},
    {'timestamp': '20:38:12', 'action': 'SELL', 'confidence': 0.72, 'price': 2413.45},
    {'timestamp': '20:37:58', 'action': 'HOLD', 'confidence': 0.02, 'price': 2412.89}
 ]
 # Add signals to dashboard
 for signal_data in test_signals:
    test_signal = {
        'timestamp': signal_data['timestamp'],
        'action': signal_data['action'],
        'confidence': signal_data['confidence'],
        'price': signal_data['price'],
        'symbol': 'ETH/USDT',
        'executed': False,
        'blocked': True,
        'manual': False,
        'model': 'TEST'
    }
    dashboard._process_dashboard_signal(test_signal)
 print(f'After adding {len(test_signals)} signals: {len(dashboard.recent_decisions)}')
 # Test with larger batch to verify new limits
 print('\nAdding 50 more signals to test preservation...')
 for i in range(50):
    test_signal = {
        'timestamp': f'20:3{i//10}:{i%60:02d}',
        'action': 'HOLD' if i % 3 == 0 else ('BUY' if i % 2 == 0 else 'SELL'),
        'confidence': 0.01 + (i * 0.01),
        'price': 2420.0 + i,
        'symbol': 'ETH/USDT',
        'executed': False,
        'blocked': True,
        'manual': False,
        'model': 'BATCH_TEST'
    }
    dashboard._process_dashboard_signal(test_signal)
 print(f'After adding 50 more signals: {len(dashboard.recent_decisions)}')
 # Display recent signals
 print('\nRecent signals (last 10):')
 for signal in dashboard.recent_decisions[-10:]:
    timestamp = dashboard._get_signal_attribute(signal, 'timestamp', 'Unknown')
    action = dashboard._get_signal_attribute(signal, 'action', 'UNKNOWN')
    confidence = dashboard._get_signal_attribute(signal, 'confidence', 0)
    price = dashboard._get_signal_attribute(signal, 'price', 0)
    print(f'  {timestamp} {action}({confidence*100:.1f}%) ${price:.2f}')
 # Test cleanup behavior with tick cache
 print('\nTesting tick cache cleanup behavior...')
 dashboard.tick_cache = [
    {'datetime': time.time() - 3600, 'symbol': 'ETHUSDT', 'price': 2400.0},  # 1 hour ago
    {'datetime': time.time() - 1800, 'symbol': 'ETHUSDT', 'price': 2410.0},  # 30 min ago
    {'datetime': time.time() - 900, 'symbol': 'ETHUSDT', 'price': 2420.0},   # 15 min ago
 ]
 # This should NOT clear signals aggressively anymore
 signals_before = len(dashboard.recent_decisions)
 dashboard._clear_old_signals_for_tick_range()
 signals_after = len(dashboard.recent_decisions)
 print(f'Signals before cleanup: {signals_before}')
 print(f'Signals after cleanup: {signals_after}')
 print(f'Signals preserved: {signals_after}/{signals_before} ({(signals_after/signals_before)*100:.1f}%)')
 print('\n✅ Signal preservation test completed!')
 print('Changes made:')
 print('- Increased recent_decisions limit from 20/50 to 200')
 print('- Made tick cache cleanup much more conservative')
 print('- Only clears when >500 signals and removes >20% of old data')
 print('- Extended time range for signal preservation') 
--- a/test_simplified_architecture.py
+++ b/test_simplified_architecture.py
@ -1,277 +0,0 @@
 #!/usr/bin/env python3
 """
 Test Simplified Architecture
 Demonstrates the new simplified data architecture:
 - Simple cache instead of FIFO queues
 - Smart data updates with minimal API calls
 - Efficient tick-based candle construction
 """
 import time
 from datetime import datetime
 from core.data_provider import DataProvider
 from core.simplified_data_integration import SimplifiedDataIntegration
 from core.data_cache import get_data_cache
 def test_simplified_cache():
    """Test the simplified cache system"""
    print("=== Testing Simplified Cache System ===")
    try:
        cache = get_data_cache()
        # Test basic cache operations
        print("1. Testing basic cache operations:")
        # Update cache with some data
        test_data = {'price': 3500.0, 'volume': 1000.0}
        success = cache.update('test_data', 'ETH/USDT', test_data, 'test')
        print(f"  Cache update: {'✅' if success else '❌'}")
        # Retrieve data
        retrieved = cache.get('test_data', 'ETH/USDT')
        print(f"  Data retrieval: {'✅' if retrieved == test_data else '❌'}")
        # Test metadata
        entry = cache.get_with_metadata('test_data', 'ETH/USDT')
        if entry:
            print(f"  Metadata: source={entry.source}, version={entry.version}")
        # Test data existence check
        has_data = cache.has_data('test_data', 'ETH/USDT')
        print(f"  Data existence check: {'✅' if has_data else '❌'}")
        # Test status
        status = cache.get_status()
        print(f"  Cache status: {len(status)} data types")
        return True
    except Exception as e:
        print(f"❌ Cache test failed: {e}")
        return False
 def test_smart_data_updater():
    """Test the smart data updater"""
    print("\n=== Testing Smart Data Updater ===")
    try:
        data_provider = DataProvider()
        symbols = ['ETH/USDT', 'BTC/USDT']
        # Create simplified integration
        integration = SimplifiedDataIntegration(data_provider, symbols)
        print("1. Starting data integration...")
        integration.start()
        # Wait for initial data load
        print("2. Waiting for initial data load (10 seconds)...")
        time.sleep(10)
        # Check cache status
        print("3. Checking cache status:")
        status = integration.get_cache_status()
        cache_status = status.get('cache_status', {})
        for data_type, symbols_data in cache_status.items():
            print(f"  {data_type}:")
            for symbol, info in symbols_data.items():
                age = info.get('age_seconds', 0)
                has_data = info.get('has_data', False)
                source = info.get('source', 'unknown')
                status_icon = '✅' if has_data and age < 300 else '❌'
                print(f"    {symbol}: {status_icon} age={age:.1f}s, source={source}")
        # Test current price
        print("4. Testing current price retrieval:")
        for symbol in symbols:
            price = integration.get_current_price(symbol)
            if price:
                print(f"  {symbol}: ${price:.2f} ✅")
            else:
                print(f"  {symbol}: No price data ❌")
        # Test data sufficiency
        print("5. Testing data sufficiency:")
        for symbol in symbols:
            sufficient = integration.has_sufficient_data(symbol)
            print(f"  {symbol}: {'✅ Sufficient' if sufficient else '❌ Insufficient'}")
        integration.stop()
        return True
    except Exception as e:
        print(f"❌ Smart data updater test failed: {e}")
        return False
 def test_base_data_input_building():
    """Test BaseDataInput building with simplified architecture"""
    print("\n=== Testing BaseDataInput Building ===")
    try:
        data_provider = DataProvider()
        symbols = ['ETH/USDT', 'BTC/USDT']
        integration = SimplifiedDataIntegration(data_provider, symbols)
        integration.start()
        # Wait for data
        print("1. Loading data...")
        time.sleep(8)
        # Test BaseDataInput building
        print("2. Testing BaseDataInput building:")
        for symbol in symbols:
            try:
                base_data = integration.build_base_data_input(symbol)
                if base_data:
                    features = base_data.get_feature_vector()
                    print(f"  {symbol}: ✅ BaseDataInput built")
                    print(f"    Feature vector size: {len(features)}")
                    print(f"    OHLCV 1s: {len(base_data.ohlcv_1s)} bars")
                    print(f"    OHLCV 1m: {len(base_data.ohlcv_1m)} bars")
                    print(f"    OHLCV 1h: {len(base_data.ohlcv_1h)} bars")
                    print(f"    OHLCV 1d: {len(base_data.ohlcv_1d)} bars")
                    print(f"    BTC reference: {len(base_data.btc_ohlcv_1s)} bars")
                    print(f"    Technical indicators: {len(base_data.technical_indicators)}")
                    # Validate feature vector size
                    if len(features) == 7850:
                        print(f"    ✅ Feature vector has correct size")
                    else:
                        print(f"    ⚠️ Feature vector size: {len(features)} (expected 7850)")
                    # Test validation
                    is_valid = base_data.validate()
                    print(f"    Validation: {'✅ PASSED' if is_valid else '❌ FAILED'}")
                else:
                    print(f"  {symbol}: ❌ Failed to build BaseDataInput")
            except Exception as e:
                print(f"  {symbol}: ❌ Error - {e}")
        integration.stop()
        return True
    except Exception as e:
        print(f"❌ BaseDataInput test failed: {e}")
        return False
 def test_tick_simulation():
    """Test tick data processing simulation"""
    print("\n=== Testing Tick Data Processing ===")
    try:
        data_provider = DataProvider()
        symbols = ['ETH/USDT']
        integration = SimplifiedDataIntegration(data_provider, symbols)
        integration.start()
        # Wait for initial setup
        time.sleep(3)
        print("1. Simulating tick data...")
        # Simulate some tick data
        base_price = 3500.0
        for i in range(20):
            price = base_price + (i * 0.1) - 1.0  # Small price movements
            volume = 10.0 + (i * 0.5)
            # Add tick data
            integration.data_updater.add_tick('ETH/USDT', price, volume)
            time.sleep(0.1)  # 100ms between ticks
        print("2. Waiting for tick processing...")
        time.sleep(12)  # Wait for 1s candle construction
        # Check if 1s candle was built from ticks
        cache = get_data_cache()
        ohlcv_1s = cache.get('ohlcv_1s', 'ETH/USDT')
        if ohlcv_1s:
            print(f"3. ✅ 1s candle built from ticks:")
            print(f"   Price: {ohlcv_1s.close:.2f}")
            print(f"   Volume: {ohlcv_1s.volume:.2f}")
            print(f"   Source: tick_constructed")
        else:
            print(f"3. ❌ No 1s candle built from ticks")
        integration.stop()
        return ohlcv_1s is not None
    except Exception as e:
        print(f"❌ Tick simulation test failed: {e}")
        return False
 def test_efficiency_comparison():
    """Compare efficiency with old FIFO queue approach"""
    print("\n=== Efficiency Comparison ===")
    print("Simplified Architecture Benefits:")
    print("✅ Single cache entry per data type (vs. 500-item queues)")
    print("✅ Unordered updates supported")
    print("✅ Minimal API calls (1m/minute, 1h/hour vs. every second)")
    print("✅ Smart tick-based 1s candle construction")
    print("✅ Extensible for new data types")
    print("✅ Thread-safe with minimal locking")
    print("✅ Historical data loaded once at startup")
    print("✅ Automatic fallback strategies")
    print("\nMemory Usage Comparison:")
    print("Old: ~500 OHLCV bars × 4 timeframes × 2 symbols = ~4000 objects")
    print("New: ~1 current bar × 4 timeframes × 2 symbols = ~8 objects")
    print("Reduction: ~99.8% memory usage for current data")
    print("\nAPI Call Comparison:")
    print("Old: Continuous polling every second for all timeframes")
    print("New: 1s from ticks, 1m every minute, 1h every hour, 1d daily")
    print("Reduction: ~95% fewer API calls")
    return True
 def main():
    """Run all simplified architecture tests"""
    print("=== Simplified Data Architecture Test Suite ===")
    tests = [
        ("Simplified Cache", test_simplified_cache),
        ("Smart Data Updater", test_smart_data_updater),
        ("BaseDataInput Building", test_base_data_input_building),
        ("Tick Data Processing", test_tick_simulation),
        ("Efficiency Comparison", test_efficiency_comparison)
    ]
    passed = 0
    total = len(tests)
    for test_name, test_func in tests:
        print(f"\n{'='*60}")
        try:
            if test_func():
                passed += 1
                print(f"✅ {test_name}: PASSED")
            else:
                print(f"❌ {test_name}: FAILED")
        except Exception as e:
            print(f"❌ {test_name}: ERROR - {e}")
    print(f"\n{'='*60}")
    print(f"=== Test Results: {passed}/{total} passed ===")
    if passed == total:
        print("\n🎉 ALL TESTS PASSED!")
        print("✅ Simplified architecture is working correctly")
        print("✅ Much more efficient than FIFO queues")
        print("✅ Ready for production use")
    else:
        print(f"\n⚠️ {total - passed} tests failed")
        print("Check individual test results above")
 if __name__ == "__main__":
    main()
--- a/test_simplified_data_provider.py
+++ b/test_simplified_data_provider.py
@ -1,60 +0,0 @@
 #!/usr/bin/env python3
 """
 Test script for the simplified data provider
 """
 import time
 import logging
 from core.data_provider import DataProvider
 # Set up logging
 logging.basicConfig(level=logging.INFO)
 logger = logging.getLogger(__name__)
 def test_data_provider():
    """Test the simplified data provider"""
    logger.info("Testing simplified data provider...")
    # Initialize data provider
    dp = DataProvider()
    # Wait for initial data load
    logger.info("Waiting for initial data load...")
    time.sleep(10)
    # Check health
    health = dp.health_check()
    logger.info(f"Health check: {health}")
    # Get cached data summary
    summary = dp.get_cached_data_summary()
    logger.info(f"Cached data summary: {summary}")
    # Test getting historical data (should be from cache only)
    for symbol in ['ETH/USDT', 'BTC/USDT']:
        for timeframe in ['1s', '1m', '1h', '1d']:
            data = dp.get_historical_data(symbol, timeframe, limit=10)
            if data is not None and not data.empty:
                logger.info(f"{symbol} {timeframe}: {len(data)} candles, latest price: {data.iloc[-1]['close']}")
            else:
                logger.warning(f"{symbol} {timeframe}: No data available")
    # Test current prices
    for symbol in ['ETH/USDT', 'BTC/USDT']:
        price = dp.get_current_price(symbol)
        logger.info(f"Current price for {symbol}: {price}")
    # Wait and check if data is being updated
    logger.info("Waiting 30 seconds to check data updates...")
    time.sleep(30)
    # Check data again
    summary2 = dp.get_cached_data_summary()
    logger.info(f"Updated cached data summary: {summary2}")
    # Stop data maintenance
    dp.stop_automatic_data_maintenance()
    logger.info("Test completed")
 if __name__ == "__main__":
    test_data_provider()
--- a/test_standardized_data_provider.py
+++ b/test_standardized_data_provider.py
@ -1,128 +0,0 @@
 """
 Test script for StandardizedDataProvider
 This script tests the standardized BaseDataInput functionality
 """
 import sys
 import os
 sys.path.append(os.path.dirname(os.path.abspath(__file__)))
 import logging
 from datetime import datetime
 from core.standardized_data_provider import StandardizedDataProvider
 from core.data_models import create_model_output
 # Set up logging
 logging.basicConfig(level=logging.INFO)
 logger = logging.getLogger(__name__)
 def test_standardized_data_provider():
    """Test the StandardizedDataProvider functionality"""
    print("Testing StandardizedDataProvider...")
    # Initialize the provider
    symbols = ['ETH/USDT', 'BTC/USDT']
    timeframes = ['1s', '1m', '1h', '1d']
    provider = StandardizedDataProvider(symbols=symbols, timeframes=timeframes)
    # Test getting BaseDataInput
    print("\n1. Testing BaseDataInput creation...")
    base_input = provider.get_base_data_input('ETH/USDT')
    if base_input is None:
        print("❌ BaseDataInput is None - this is expected if no historical data is available")
        print("   The provider needs real market data to create BaseDataInput")
        # Test with real data only
        print("\n2. Testing data structures...")
        # Test ModelOutput creation
        model_output = create_model_output(
            model_type='cnn',
            model_name='test_cnn',
            symbol='ETH/USDT',
            action='BUY',
            confidence=0.75,
            metadata={'test': True}
        )
        print(f"✅ Created ModelOutput: {model_output.model_type} - {model_output.predictions['action']} ({model_output.confidence})")
        # Test storing model output
        provider.store_model_output(model_output)
        stored_outputs = provider.get_model_outputs('ETH/USDT')
        if 'test_cnn' in stored_outputs:
            print("✅ Model output storage and retrieval working")
        else:
            print("❌ Model output storage failed")
    else:
        print("✅ BaseDataInput created successfully!")
        print(f"   Symbol: {base_input.symbol}")
        print(f"   Timestamp: {base_input.timestamp}")
        print(f"   OHLCV 1s frames: {len(base_input.ohlcv_1s)}")
        print(f"   OHLCV 1m frames: {len(base_input.ohlcv_1m)}")
        print(f"   OHLCV 1h frames: {len(base_input.ohlcv_1h)}")
        print(f"   OHLCV 1d frames: {len(base_input.ohlcv_1d)}")
        print(f"   BTC 1s frames: {len(base_input.btc_ohlcv_1s)}")
        print(f"   COB data available: {base_input.cob_data is not None}")
        print(f"   Technical indicators: {len(base_input.technical_indicators)}")
        print(f"   Pivot points: {len(base_input.pivot_points)}")
        print(f"   Last predictions: {len(base_input.last_predictions)}")
        # Test feature vector creation
        try:
            feature_vector = base_input.get_feature_vector()
            print(f"✅ Feature vector created: shape {feature_vector.shape}")
        except Exception as e:
            print(f"❌ Feature vector creation failed: {e}")
        # Test validation
        is_valid = base_input.validate()
        print(f"✅ BaseDataInput validation: {'PASSED' if is_valid else 'FAILED'}")
    print("\n3. Testing data provider capabilities...")
    # Test historical data fetching
    try:
        eth_data = provider.get_historical_data('ETH/USDT', '1h', 10)
        if eth_data is not None and not eth_data.empty:
            print(f"✅ Historical data available: {len(eth_data)} bars for ETH/USDT 1h")
        else:
            print("⚠️  No historical data available - this is normal if APIs are not accessible")
    except Exception as e:
        print(f"⚠️  Historical data fetch error: {e}")
    print("\n4. Testing COB data functionality...")
    # Test COB data creation
    try:
        # Set a mock current price for testing
        provider.current_prices['ETHUSDT'] = 3000.0
        cob_data = provider._get_cob_data('ETH/USDT', datetime.now())
        if cob_data:
            print(f"✅ COB data created successfully")
            print(f"   Current price: ${cob_data.current_price}")
            print(f"   Bucket size: ${cob_data.bucket_size}")
            print(f"   Price buckets: {len(cob_data.price_buckets)}")
            print(f"   MA 1s imbalance: {len(cob_data.ma_1s_imbalance)} buckets")
            print(f"   MA 5s imbalance: {len(cob_data.ma_5s_imbalance)} buckets")
        else:
            print("⚠️  COB data creation returned None")
    except Exception as e:
        print(f"❌ COB data creation error: {e}")
    print("\n✅ StandardizedDataProvider test completed!")
    print("\nNext steps:")
    print("1. Integrate with real market data APIs")
    print("2. Connect to actual COB provider")
    print("3. Test with live data streams")
    print("4. Integrate with model training pipelines")
 if __name__ == "__main__":
    test_standardized_data_provider()
--- a/test_timezone_handling.py
+++ b/test_timezone_handling.py
@ -1,153 +0,0 @@
 #!/usr/bin/env python3
 """
 Test script to verify timezone handling in data provider
 """
 import time
 import logging
 import pandas as pd
 from datetime import datetime, timezone
 from core.data_provider import DataProvider
 # Set up logging
 logging.basicConfig(level=logging.INFO)
 logger = logging.getLogger(__name__)
 def test_timezone_handling():
    """Test timezone handling in data provider"""
    logger.info("Testing timezone handling...")
    # Initialize data provider
    dp = DataProvider()
    # Wait for initial data load
    logger.info("Waiting for initial data load...")
    time.sleep(15)
    # Test 1: Check timezone info in cached data
    logger.info("\n=== Test 1: Timezone Info in Cached Data ===")
    for symbol in ['ETH/USDT', 'BTC/USDT']:
        for timeframe in ['1s', '1m', '1h', '1d']:
            if symbol in dp.cached_data and timeframe in dp.cached_data[symbol]:
                df = dp.cached_data[symbol][timeframe]
                if not df.empty:
                    # Check if index has timezone info
                    has_tz = df.index.tz is not None
                    tz_info = df.index.tz if has_tz else "No timezone"
                    # Get first and last timestamps
                    first_ts = df.index[0]
                    last_ts = df.index[-1]
                    logger.info(f"{symbol} {timeframe}:")
                    logger.info(f"  Timezone: {tz_info}")
                    logger.info(f"  First: {first_ts}")
                    logger.info(f"  Last: {last_ts}")
                    # Check for gaps (only for timeframes with enough data)
                    if len(df) > 10:
                        # Calculate expected time difference
                        if timeframe == '1s':
                            expected_diff = pd.Timedelta(seconds=1)
                        elif timeframe == '1m':
                            expected_diff = pd.Timedelta(minutes=1)
                        elif timeframe == '1h':
                            expected_diff = pd.Timedelta(hours=1)
                        elif timeframe == '1d':
                            expected_diff = pd.Timedelta(days=1)
                        # Check for large gaps
                        time_diffs = df.index.to_series().diff()
                        large_gaps = time_diffs[time_diffs > expected_diff * 2]
                        if not large_gaps.empty:
                            logger.warning(f"  Found {len(large_gaps)} large gaps:")
                            for gap_time, gap_size in large_gaps.head(3).items():
                                logger.warning(f"    Gap at {gap_time}: {gap_size}")
                        else:
                            logger.info(f"  No significant gaps found")
                else:
                    logger.info(f"{symbol} {timeframe}: No data")
    # Test 2: Compare with current UTC time
    logger.info("\n=== Test 2: Compare with Current UTC Time ===")
    current_utc = datetime.now(timezone.utc)
    logger.info(f"Current UTC time: {current_utc}")
    for symbol in ['ETH/USDT', 'BTC/USDT']:
        # Get latest 1m data
        if symbol in dp.cached_data and '1m' in dp.cached_data[symbol]:
            df = dp.cached_data[symbol]['1m']
            if not df.empty:
                latest_ts = df.index[-1]
                # Convert to UTC if it has timezone info
                if latest_ts.tz is not None:
                    latest_utc = latest_ts.tz_convert('UTC')
                else:
                    # Assume it's already UTC if no timezone
                    latest_utc = latest_ts.replace(tzinfo=timezone.utc)
                time_diff = current_utc - latest_utc
                logger.info(f"{symbol} latest data:")
                logger.info(f"  Timestamp: {latest_ts}")
                logger.info(f"  UTC: {latest_utc}")
                logger.info(f"  Age: {time_diff}")
                # Check if data is reasonably fresh (within 1 hour)
                if time_diff.total_seconds() < 3600:
                    logger.info(f"  ✅ Data is fresh")
                else:
                    logger.warning(f"  ⚠️ Data is stale (>{time_diff})")
    # Test 3: Check data continuity
    logger.info("\n=== Test 3: Data Continuity Check ===")
    for symbol in ['ETH/USDT', 'BTC/USDT']:
        if symbol in dp.cached_data and '1h' in dp.cached_data[symbol]:
            df = dp.cached_data[symbol]['1h']
            if len(df) > 24:  # At least 24 hours of data
                # Get last 24 hours
                recent_df = df.tail(24)
                # Check for 3-hour gaps (the reported issue)
                time_diffs = recent_df.index.to_series().diff()
                three_hour_gaps = time_diffs[time_diffs >= pd.Timedelta(hours=3)]
                logger.info(f"{symbol} 1h data (last 24 candles):")
                logger.info(f"  Time range: {recent_df.index[0]} to {recent_df.index[-1]}")
                if not three_hour_gaps.empty:
                    logger.warning(f"  ❌ Found {len(three_hour_gaps)} gaps >= 3 hours:")
                    for gap_time, gap_size in three_hour_gaps.items():
                        logger.warning(f"    {gap_time}: {gap_size}")
                else:
                    logger.info(f"  ✅ No 3+ hour gaps found")
                # Show time differences
                logger.info(f"  Time differences (last 5):")
                for i, (ts, diff) in enumerate(time_diffs.tail(5).items()):
                    if pd.notna(diff):
                        logger.info(f"    {ts}: {diff}")
    # Test 4: Manual timezone conversion test
    logger.info("\n=== Test 4: Manual Timezone Conversion Test ===")
    # Create test timestamps
    utc_now = datetime.now(timezone.utc)
    local_now = datetime.now()
    logger.info(f"UTC now: {utc_now}")
    logger.info(f"Local now: {local_now}")
    logger.info(f"Difference: {utc_now - local_now.replace(tzinfo=timezone.utc)}")
    # Test pandas timezone handling
    test_ts = pd.Timestamp.now(tz='UTC')
    logger.info(f"Pandas UTC timestamp: {test_ts}")
    # Clean shutdown
    logger.info("\n=== Shutting Down ===")
    dp.stop_automatic_data_maintenance()
    logger.info("Timezone handling test completed")
 if __name__ == "__main__":
    test_timezone_handling()
--- a/test_timezone_with_data.py
+++ b/test_timezone_with_data.py
@ -1,136 +0,0 @@
 #!/usr/bin/env python3
 """
 Test Timezone Fix with Data Fetching
 This script tests timezone conversion by actually fetching data and checking timestamps.
 """
 import asyncio
 import pandas as pd
 from datetime import datetime
 from core.data_provider import DataProvider
 async def test_timezone_with_data():
    """Test timezone conversion with actual data fetching"""
    print("=== Testing Timezone Fix with Data Fetching ===")
    # Initialize data provider
    print("1. Initializing data provider...")
    data_provider = DataProvider()
    # Wait for initialization
    await asyncio.sleep(2)
    # Test direct Binance API call
    print("\n2. Testing direct Binance API call:")
    try:
        # Call the internal Binance fetch method directly
        df = data_provider._fetch_from_binance('ETH/USDT', '1h', 5)
        if df is not None and not df.empty:
            print(f"   ✅ Got {len(df)} candles from Binance API")
            # Check timezone
            if 'timestamp' in df.columns:
                first_timestamp = df['timestamp'].iloc[0]
                last_timestamp = df['timestamp'].iloc[-1]
                print(f"   First timestamp: {first_timestamp}")
                print(f"   Last timestamp: {last_timestamp}")
                # Check if timezone is Europe/Sofia
                if hasattr(first_timestamp, 'tz') and first_timestamp.tz is not None:
                    timezone_str = str(first_timestamp.tz)
                    print(f"   Timezone: {timezone_str}")
                    if 'Europe/Sofia' in timezone_str or 'EET' in timezone_str or 'EEST' in timezone_str:
                        print(f"   ✅ Timezone is correct: {timezone_str}")
                    else:
                        print(f"   ❌ Timezone is incorrect: {timezone_str}")
                # Show UTC offset
                if hasattr(first_timestamp, 'utcoffset') and first_timestamp.utcoffset() is not None:
                    offset_hours = first_timestamp.utcoffset().total_seconds() / 3600
                    print(f"   UTC offset: {offset_hours:+.0f} hours")
                    if offset_hours == 2 or offset_hours == 3:  # EET (+2) or EEST (+3)
                        print("   ✅ UTC offset is correct for Europe/Sofia")
                    else:
                        print(f"   ❌ UTC offset is incorrect: {offset_hours:+.0f} hours")
                # Compare with UTC time
                print("\n   Timestamp comparison:")
                for i in range(min(2, len(df))):
                    row = df.iloc[i]
                    local_time = row['timestamp']
                    utc_time = local_time.astimezone(pd.Timestamp.now(tz='UTC').tz)
                    print(f"     Local (Sofia): {local_time}")
                    print(f"     UTC:           {utc_time}")
                    print(f"     Difference:    {(local_time - utc_time).total_seconds() / 3600:+.0f} hours")
                    print()
            else:
                print("   ❌ No timestamp column found")
        else:
            print("   ❌ No data returned from Binance API")
    except Exception as e:
        print(f"   ❌ Error fetching from Binance: {e}")
    # Test MEXC API call as well
    print("\n3. Testing MEXC API call:")
    try:
        df = data_provider._fetch_from_mexc('ETH/USDT', '1h', 3)
        if df is not None and not df.empty:
            print(f"   ✅ Got {len(df)} candles from MEXC API")
            # Check timezone
            if 'timestamp' in df.columns:
                first_timestamp = df['timestamp'].iloc[0]
                print(f"   First timestamp: {first_timestamp}")
                # Check timezone
                if hasattr(first_timestamp, 'tz') and first_timestamp.tz is not None:
                    timezone_str = str(first_timestamp.tz)
                    print(f"   Timezone: {timezone_str}")
                    if 'Europe/Sofia' in timezone_str or 'EET' in timezone_str or 'EEST' in timezone_str:
                        print(f"   ✅ MEXC timezone is correct: {timezone_str}")
                    else:
                        print(f"   ❌ MEXC timezone is incorrect: {timezone_str}")
                # Show UTC offset
                if hasattr(first_timestamp, 'utcoffset') and first_timestamp.utcoffset() is not None:
                    offset_hours = first_timestamp.utcoffset().total_seconds() / 3600
                    print(f"   UTC offset: {offset_hours:+.0f} hours")
        else:
            print("   ❌ No data returned from MEXC API")
    except Exception as e:
        print(f"   ❌ Error fetching from MEXC: {e}")
    # Show current timezone info
    print(f"\n4. Current timezone information:")
    import pytz
    sofia_tz = pytz.timezone('Europe/Sofia')
    current_sofia = datetime.now(sofia_tz)
    current_utc = datetime.now(pytz.UTC)
    print(f"   Current Sofia time: {current_sofia}")
    print(f"   Current UTC time:   {current_utc}")
    print(f"   Time difference:    {(current_sofia - current_utc).total_seconds() / 3600:+.0f} hours")
    # Check if it's summer time (EEST) or winter time (EET)
    offset_hours = current_sofia.utcoffset().total_seconds() / 3600
    if offset_hours == 3:
        print("   ✅ Currently in EEST (Eastern European Summer Time)")
    elif offset_hours == 2:
        print("   ✅ Currently in EET (Eastern European Time)")
    else:
        print(f"   ❌ Unexpected offset: {offset_hours:+.0f} hours")
    print("\n✅ Timezone fix test with data completed!")
 if __name__ == "__main__":
    asyncio.run(test_timezone_with_data())
--- a/test_universal_model_toggles.py
+++ b/test_universal_model_toggles.py
@ -1,150 +0,0 @@
 #!/usr/bin/env python3
 """
 Test script for the Universal Model Toggle System
 This script demonstrates how the new universal model toggle system works
 with any model, not just hardcoded ones.
 """
 import sys
 import os
 import logging
 from datetime import datetime
 # Add the project root to the path
 sys.path.append(os.path.dirname(os.path.abspath(__file__)))
 # Setup logging
 logging.basicConfig(level=logging.INFO)
 logger = logging.getLogger(__name__)
 def test_universal_model_toggles():
    """Test the universal model toggle system"""
    try:
        from core.orchestrator import TradingOrchestrator
        from core.data_provider import DataProvider
        from models import ModelInterface, get_model_registry
        logger.info("🧪 Testing Universal Model Toggle System")
        # Initialize components
        data_provider = DataProvider()
        orchestrator = TradingOrchestrator(data_provider=data_provider)
        # Test 1: Check existing models
        logger.info("\n📋 Test 1: Checking existing models")
        existing_models = orchestrator.get_all_registered_models()
        logger.info(f"Found {len(existing_models)} existing models: {list(existing_models.keys())}")
        # Test 2: Add a new model dynamically
        logger.info("\n➕ Test 2: Adding new model dynamically")
        class TestModel(ModelInterface):
            def __init__(self):
                super().__init__("test_model")
            def predict(self, data):
                return {"action": "TEST", "confidence": 0.85}
        test_model = TestModel()
        success = orchestrator.register_model_dynamically("test_model", test_model)
        logger.info(f"Dynamic model registration: {'✅ SUCCESS' if success else '❌ FAILED'}")
        # Test 3: Check toggle states
        logger.info("\n🔄 Test 3: Testing toggle states")
        # Test with existing model
        dqn_state = orchestrator.get_model_toggle_state("dqn")
        logger.info(f"DQN toggle state: {dqn_state}")
        # Test with new model
        test_model_state = orchestrator.get_model_toggle_state("test_model")
        logger.info(f"Test model toggle state: {test_model_state}")
        # Test 4: Update toggle states
        logger.info("\n⚙️ Test 4: Updating toggle states")
        # Disable inference for test model
        orchestrator.set_model_toggle_state("test_model", inference_enabled=False)
        updated_state = orchestrator.get_model_toggle_state("test_model")
        logger.info(f"Updated test model state: {updated_state}")
        # Test 5: Add another model without interface
        logger.info("\n➕ Test 5: Adding model without interface")
        orchestrator.set_model_toggle_state("custom_transformer", inference_enabled=True, training_enabled=True)
        transformer_state = orchestrator.get_model_toggle_state("custom_transformer")
        logger.info(f"Custom transformer state: {transformer_state}")
        # Test 6: Check all models after additions
        logger.info("\n📋 Test 6: Final model count")
        final_models = orchestrator.get_all_registered_models()
        logger.info(f"Final model count: {len(final_models)}")
        for model_name, model_info in final_models.items():
            toggle_state = orchestrator.get_model_toggle_state(model_name)
            logger.info(f"  - {model_name}: inf={toggle_state['inference_enabled']}, train={toggle_state['training_enabled']}")
        logger.info("\n✅ Universal Model Toggle System test completed successfully!")
        return True
    except Exception as e:
        logger.error(f"❌ Test failed: {e}")
        return False
 def test_dashboard_integration():
    """Test dashboard integration with universal toggles"""
    try:
        logger.info("\n🖥️ Testing Dashboard Integration")
        from web.clean_dashboard import CleanTradingDashboard
        from core.orchestrator import TradingOrchestrator
        from core.data_provider import DataProvider
        # Initialize components
        data_provider = DataProvider()
        orchestrator = TradingOrchestrator(data_provider=data_provider)
        # Add some test models
        orchestrator.set_model_toggle_state("test_model_1", inference_enabled=True, training_enabled=False)
        orchestrator.set_model_toggle_state("test_model_2", inference_enabled=False, training_enabled=True)
        # Initialize dashboard (this will test the universal callback setup)
        dashboard = CleanTradingDashboard(
            data_provider=data_provider,
            orchestrator=orchestrator
        )
        # Test adding model dynamically through dashboard
        success = dashboard.add_model_dynamically("dynamic_test_model")
        logger.info(f"Dashboard dynamic model addition: {'✅ SUCCESS' if success else '❌ FAILED'}")
        # Check available models
        available_models = dashboard._get_available_models()
        logger.info(f"Dashboard sees {len(available_models)} models: {list(available_models.keys())}")
        logger.info("✅ Dashboard integration test completed!")
        return True
    except Exception as e:
        logger.error(f"❌ Dashboard integration test failed: {e}")
        return False
 if __name__ == "__main__":
    logger.info("🚀 Starting Universal Model Toggle System Tests")
    logger.info("=" * 60)
    # Run tests
    test1_success = test_universal_model_toggles()
    test2_success = test_dashboard_integration()
    # Summary
    logger.info("\n" + "=" * 60)
    logger.info("📊 TEST SUMMARY")
    logger.info(f"Universal Toggle System: {'✅ PASS' if test1_success else '❌ FAIL'}")
    logger.info(f"Dashboard Integration: {'✅ PASS' if test2_success else '❌ FAIL'}")
    if test1_success and test2_success:
        logger.info("🎉 ALL TESTS PASSED! Universal model toggle system is working correctly.")
        sys.exit(0)
    else:
        logger.error("❌ Some tests failed. Check the logs above for details.")
        sys.exit(1)
--- a/validate_training_system.py
+++ b/validate_training_system.py
@ -1,409 +0,0 @@
 #!/usr/bin/env python3
 """
 Training System Validation
 This script validates that the core training system is working correctly:
 1. Data provider is supplying quality data
 2. Models can be loaded and make predictions
 3. State building is working (13,400 features)
 4. Reward calculation is functioning
 5. Training loop can run without errors
 Focus: Core functionality validation, not performance optimization
 """
 import os
 import sys
 import asyncio
 import logging
 import numpy as np
 from datetime import datetime
 from pathlib import Path
 # Add project root to path
 project_root = Path(__file__).parent
 sys.path.insert(0, str(project_root))
 from core.config import setup_logging, get_config
 from core.data_provider import DataProvider
 from core.orchestrator import TradingOrchestrator
 from core.trading_executor import TradingExecutor
 # Setup logging
 setup_logging()
 logger = logging.getLogger(__name__)
 class TrainingSystemValidator:
    """
    Validates core training system functionality
    """
    def __init__(self):
        """Initialize validator"""
        self.config = get_config()
        self.validation_results = {
            'data_provider': False,
            'orchestrator': False,
            'state_building': False,
            'reward_calculation': False,
            'model_loading': False,
            'training_loop': False
        }
        # Components
        self.data_provider = None
        self.orchestrator = None
        self.trading_executor = None
        logger.info("Training System Validator initialized")
    async def run_validation(self):
        """Run complete validation suite"""
        logger.info("=" * 60)
        logger.info("TRAINING SYSTEM VALIDATION")
        logger.info("=" * 60)
        try:
            # 1. Validate Data Provider
            await self._validate_data_provider()
            # 2. Validate Orchestrator
            await self._validate_orchestrator()
            # 3. Validate State Building
            await self._validate_state_building()
            # 4. Validate Reward Calculation
            await self._validate_reward_calculation()
            # 5. Validate Model Loading
            await self._validate_model_loading()
            # 6. Validate Training Loop
            await self._validate_training_loop()
            # Generate final report
            self._generate_validation_report()
        except Exception as e:
            logger.error(f"Validation failed: {e}")
            import traceback
            logger.error(traceback.format_exc())
    async def _validate_data_provider(self):
        """Validate data provider functionality"""
        try:
            logger.info("[1/6] Validating Data Provider...")
            # Initialize data provider
            self.data_provider = DataProvider()
            # Test historical data fetching
            symbols = ['ETH/USDT', 'BTC/USDT']
            timeframes = ['1m', '1h']
            for symbol in symbols:
                for timeframe in timeframes:
                    df = self.data_provider.get_historical_data(symbol, timeframe, limit=100)
                    if df is not None and not df.empty:
                        logger.info(f"  ✓ {symbol} {timeframe}: {len(df)} candles")
                    else:
                        logger.warning(f"  ✗ {symbol} {timeframe}: No data")
                        return
            # Test real-time data capabilities
            if hasattr(self.data_provider, 'start_real_time_streaming'):
                logger.info("  ✓ Real-time streaming available")
            else:
                logger.warning("  ✗ Real-time streaming not available")
            self.validation_results['data_provider'] = True
            logger.info("  ✓ Data Provider validation PASSED")
        except Exception as e:
            logger.error(f"  ✗ Data Provider validation FAILED: {e}")
            self.validation_results['data_provider'] = False
    async def _validate_orchestrator(self):
        """Validate orchestrator functionality"""
        try:
            logger.info("[2/6] Validating Orchestrator...")
            # Initialize orchestrator
            self.orchestrator = TradingOrchestrator(
                data_provider=self.data_provider,
                enhanced_rl_training=True
            )
            # Check if orchestrator has required methods
            required_methods = [
                'make_trading_decision',
                'build_comprehensive_rl_state',
                'make_coordinated_decisions'
            ]
            for method in required_methods:
                if hasattr(self.orchestrator, method):
                    logger.info(f"  ✓ Method '{method}' available")
                else:
                    logger.warning(f"  ✗ Method '{method}' missing")
                    return
            # Check model initialization
            if hasattr(self.orchestrator, 'rl_agent') and self.orchestrator.rl_agent:
                logger.info("  ✓ RL Agent initialized")
            else:
                logger.warning("  ✗ RL Agent not initialized")
            if hasattr(self.orchestrator, 'cnn_model') and self.orchestrator.cnn_model:
                logger.info("  ✓ CNN Model initialized")
            else:
                logger.warning("  ✗ CNN Model not initialized")
            self.validation_results['orchestrator'] = True
            logger.info("  ✓ Orchestrator validation PASSED")
        except Exception as e:
            logger.error(f"  ✗ Orchestrator validation FAILED: {e}")
            self.validation_results['orchestrator'] = False
    async def _validate_state_building(self):
        """Validate comprehensive state building"""
        try:
            logger.info("[3/6] Validating State Building...")
            if not self.orchestrator:
                logger.error("  ✗ Orchestrator not available")
                return
            # Test state building for ETH/USDT
            if hasattr(self.orchestrator, 'build_comprehensive_rl_state'):
                state = self.orchestrator.build_comprehensive_rl_state('ETH/USDT')
                if state is not None:
                    state_size = len(state)
                    logger.info(f"  ✓ ETH state built: {state_size} features")
                    # Check if we're getting the expected 13,400 features
                    if state_size == 13400:
                        logger.info("  ✓ Perfect: Exactly 13,400 features as expected")
                    elif state_size > 1000:
                        logger.info(f"  ✓ Good: {state_size} features (comprehensive)")
                    else:
                        logger.warning(f"  ⚠ Limited: Only {state_size} features")
                    # Analyze feature quality
                    non_zero_features = np.count_nonzero(state)
                    non_zero_percent = (non_zero_features / len(state)) * 100
                    logger.info(f"  ✓ Non-zero features: {non_zero_features:,} ({non_zero_percent:.1f}%)")
                    if non_zero_percent > 10:
                        logger.info("  ✓ Good feature distribution")
                    else:
                        logger.warning("  ⚠ Low feature density - may indicate data issues")
                else:
                    logger.error("  ✗ State building returned None")
                    return
            else:
                logger.error("  ✗ build_comprehensive_rl_state method not available")
                return
            self.validation_results['state_building'] = True
            logger.info("  ✓ State Building validation PASSED")
        except Exception as e:
            logger.error(f"  ✗ State Building validation FAILED: {e}")
            self.validation_results['state_building'] = False
    async def _validate_reward_calculation(self):
        """Validate reward calculation functionality"""
        try:
            logger.info("[4/6] Validating Reward Calculation...")
            if not self.orchestrator:
                logger.error("  ✗ Orchestrator not available")
                return
            # Test enhanced reward calculation if available
            if hasattr(self.orchestrator, 'calculate_enhanced_pivot_reward'):
                # Create mock data for testing
                trade_decision = {
                    'action': 'BUY',
                    'confidence': 0.75,
                    'price': 2500.0,
                    'timestamp': datetime.now()
                }
                market_data = {
                    'volatility': 0.03,
                    'order_flow_direction': 'bullish',
                    'order_flow_strength': 0.8
                }
                trade_outcome = {
                    'net_pnl': 50.0,
                    'exit_price': 2550.0
                }
                reward = self.orchestrator.calculate_enhanced_pivot_reward(
                    trade_decision, market_data, trade_outcome
                )
                if reward is not None:
                    logger.info(f"  ✓ Enhanced reward calculated: {reward:.3f}")
                else:
                    logger.warning("  ⚠ Enhanced reward calculation returned None")
            else:
                logger.warning("  ⚠ Enhanced reward calculation not available")
            # Test basic reward calculation
            # This would depend on the specific implementation
            logger.info("  ✓ Basic reward calculation available")
            self.validation_results['reward_calculation'] = True
            logger.info("  ✓ Reward Calculation validation PASSED")
        except Exception as e:
            logger.error(f"  ✗ Reward Calculation validation FAILED: {e}")
            self.validation_results['reward_calculation'] = False
    async def _validate_model_loading(self):
        """Validate model loading and checkpoints"""
        try:
            logger.info("[5/6] Validating Model Loading...")
            if not self.orchestrator:
                logger.error("  ✗ Orchestrator not available")
                return
            # Check RL Agent
            if hasattr(self.orchestrator, 'rl_agent') and self.orchestrator.rl_agent:
                logger.info("  ✓ RL Agent loaded")
                # Test prediction capability with real data
                if hasattr(self.orchestrator.rl_agent, 'predict'):
                    try:
                        # Use real state from orchestrator instead of dummy data
                        real_state = self.orchestrator._get_rl_state('ETH/USDT')
                        if real_state is not None:
                            prediction = self.orchestrator.rl_agent.predict(real_state)
                            logger.info("  ✓ RL Agent can make predictions with real data")
                        else:
                            logger.warning("  ⚠ No real state available for RL prediction test")
                    except Exception as e:
                        logger.warning(f"  ⚠ RL Agent prediction failed: {e}")
                else:
                    logger.warning("  ⚠ RL Agent predict method not available")
            else:
                logger.warning("  ⚠ RL Agent not loaded")
            # Check CNN Model
            if hasattr(self.orchestrator, 'cnn_model') and self.orchestrator.cnn_model:
                logger.info("  ✓ CNN Model loaded")
                # Test prediction capability
                if hasattr(self.orchestrator.cnn_model, 'predict'):
                    logger.info("  ✓ CNN Model can make predictions")
                else:
                    logger.warning("  ⚠ CNN Model predict method not available")
            else:
                logger.warning("  ⚠ CNN Model not loaded")
            self.validation_results['model_loading'] = True
            logger.info("  ✓ Model Loading validation PASSED")
        except Exception as e:
            logger.error(f"  ✗ Model Loading validation FAILED: {e}")
            self.validation_results['model_loading'] = False
    async def _validate_training_loop(self):
        """Validate training loop functionality"""
        try:
            logger.info("[6/6] Validating Training Loop...")
            if not self.orchestrator:
                logger.error("  ✗ Orchestrator not available")
                return
            # Test making coordinated decisions
            if hasattr(self.orchestrator, 'make_coordinated_decisions'):
                decisions = await self.orchestrator.make_coordinated_decisions()
                if decisions:
                    logger.info(f"  ✓ Coordinated decisions made: {len(decisions)} symbols")
                    for symbol, decision in decisions.items():
                        if decision:
                            logger.info(f"    - {symbol}: {decision.action} (confidence: {decision.confidence:.3f})")
                        else:
                            logger.info(f"    - {symbol}: No decision")
                else:
                    logger.warning("  ⚠ No coordinated decisions made")
            else:
                logger.warning("  ⚠ make_coordinated_decisions method not available")
            # Test individual trading decision
            if hasattr(self.orchestrator, 'make_trading_decision'):
                decision = await self.orchestrator.make_trading_decision('ETH/USDT')
                if decision:
                    logger.info(f"  ✓ Trading decision made: {decision.action} (confidence: {decision.confidence:.3f})")
                else:
                    logger.info("  ✓ No trading decision (normal behavior)")
            else:
                logger.warning("  ⚠ make_trading_decision method not available")
            self.validation_results['training_loop'] = True
            logger.info("  ✓ Training Loop validation PASSED")
        except Exception as e:
            logger.error(f"  ✗ Training Loop validation FAILED: {e}")
            self.validation_results['training_loop'] = False
    def _generate_validation_report(self):
        """Generate final validation report"""
        logger.info("=" * 60)
        logger.info("VALIDATION REPORT")
        logger.info("=" * 60)
        passed_tests = sum(1 for result in self.validation_results.values() if result)
        total_tests = len(self.validation_results)
        logger.info(f"Tests Passed: {passed_tests}/{total_tests}")
        logger.info("")
        for test_name, result in self.validation_results.items():
            status = "✓ PASS" if result else "✗ FAIL"
            logger.info(f"{test_name.replace('_', ' ').title()}: {status}")
        logger.info("")
        if passed_tests == total_tests:
            logger.info("🎉 ALL VALIDATIONS PASSED - Training system is ready!")
        elif passed_tests >= total_tests * 0.8:
            logger.info("⚠️  MOSTLY PASSED - Training system is mostly functional")
        else:
            logger.error("❌ VALIDATION FAILED - Training system needs fixes")
        logger.info("=" * 60)
        return passed_tests / total_tests
 async def main():
    """Main validation function"""
    try:
        validator = TrainingSystemValidator()
        await validator.run_validation()
    except KeyboardInterrupt:
        logger.info("Validation interrupted by user")
    except Exception as e:
        logger.error(f"Validation error: {e}")
        import traceback
        logger.error(traceback.format_exc())
 if __name__ == "__main__":
    asyncio.run(main())