gogo2/.kiro/specs/1.multi-modal-trading-system/tasks.md

# Implementation Plan

## Data Provider Backbone Enhancement

### Phase 1: Core Data Provider Enhancements

- [ ] 1. Audit and validate existing DataProvider implementation
  - Review core/data_provider.py for completeness and correctness
  - Validate 1500-candle caching is working correctly
  - Verify automatic data maintenance worker is updating properly
  - Test fallback mechanisms between Binance and MEXC
  - Document any gaps or issues found
  - _Requirements: 1.1, 1.2, 1.6_

- [ ] 1.1. Enhance COB data collection robustness
  - Fix 'NoneType' object has no attribute 'append' errors in _cob_aggregation_worker
  - Add defensive checks before accessing deque structures
  - Implement proper initialization guards to prevent duplicate COB collection starts
  - Add comprehensive error logging for COB data processing failures
  - Test COB collection under various failure scenarios
  - _Requirements: 1.3, 1.6_

- [ ] 1.2. Implement configurable COB price ranges
  - Replace hardcoded price ranges ($5 ETH, $50 BTC) with configuration
  - Add _get_price_range_for_symbol() configuration support
  - Allow per-symbol price range customization via config.yaml
  - Update COB imbalance calculations to use configurable ranges
  - Document price range selection rationale
  - _Requirements: 1.4, 1.1_

- [ ] 1.3. Validate and enhance Williams Market Structure pivot calculation
  - Review williams_market_structure.py implementation
  - Verify 5-level pivot detection is working correctly
  - Test monthly 1s data analysis for comprehensive context

  - Add unit tests for pivot point detection accuracy
  - Optimize pivot calculation performance if needed
  - _Requirements: 1.5, 2.7_

- [ ] 1.4. Implement COB heatmap matrix generation
  - Create get_cob_heatmap_matrix() method in DataProvider
  - Generate time x price matrix for visualization and model input
  - Support configurable time windows (default 300 seconds)
  - Support configurable price bucket radius (default ±10 buckets)
  - Support multiple metrics (imbalance, volume, spread)
  - Cache heatmap data for performance
  - _Requirements: 1.4, 1.1_

- [x] 1.5. Enhance EnhancedCOBWebSocket reliability
  - Review enhanced_cob_websocket.py for stability issues
  - Verify proper order book synchronization with REST snapshots
  - Test reconnection logic with exponential backoff
  - Ensure 24-hour connection limit compliance
  - Add comprehensive error handling for all WebSocket streams
  - _Requirements: 1.3, 1.6_

### Phase 2: StandardizedDataProvider Enhancements

- [ ] 2. Implement comprehensive BaseDataInput validation
  - Enhance validate() method in BaseDataInput dataclass
  - Add minimum frame count validation (100 frames per timeframe)
  - Implement data completeness scoring (0.0 to 1.0)
  - Add COB data validation (non-null, valid buckets)
  - Create detailed validation error messages
  - Prevent model inference on incomplete data (completeness < 0.8)
  - _Requirements: 1.1.2, 1.1.6_

- [ ] 2.1. Integrate COB heatmap into BaseDataInput
  - Add cob_heatmap_times, cob_heatmap_prices, cob_heatmap_values fields
  - Call get_cob_heatmap_matrix() in get_base_data_input()
  - Handle heatmap generation failures gracefully
  - Store heatmap mid_prices in market_microstructure
  - Document heatmap usage for models
  - _Requirements: 1.1.1, 1.4_

- [ ] 2.2. Enhance COB moving average calculation
  - Review _calculate_cob_moving_averages() for correctness
  - Fix bucket quantization to match COB snapshot buckets
  - Implement nearest-key matching for historical imbalance lookup
  - Add thread-safe access to cob_imbalance_history
  - Optimize MA calculation performance
  - _Requirements: 1.1.3, 1.4_

- [ ] 2.3. Implement data quality scoring system
  - Create data_quality_score() method
  - Score based on: data completeness, freshness, consistency
  - Add quality thresholds for model inference
  - Log quality metrics for monitoring
  - Provide quality breakdown in BaseDataInput
  - _Requirements: 1.1.2, 1.1.6_

- [ ] 2.4. Enhance live price fetching robustness
  - Review get_live_price_from_api() fallback chain
  - Add retry logic with exponential backoff
  - Implement circuit breaker for repeated API failures
  - Cache prices with configurable TTL (default 500ms)
  - Log price source for debugging
  - _Requirements: 1.6, 1.7_

### Phase 3: COBY Integration

- [ ] 3. Design unified interface between COBY and core DataProvider
  - Define clear boundaries between COBY and core systems
  - Create adapter layer for accessing COBY data from core
  - Design data flow for multi-exchange aggregation
  - Plan migration path for existing code
  - Document integration architecture
  - _Requirements: 1.10, 8.1_

- [ ] 3.1. Implement COBY data access adapter
  - Create COBYDataAdapter class in core/
  - Implement methods to query COBY TimescaleDB
  - Add Redis cache integration for performance
  - Support historical data retrieval from COBY
  - Handle COBY unavailability gracefully
  - _Requirements: 1.10, 8.1_

- [ ] 3.2. Integrate COBY heatmap data
  - Query COBY for multi-exchange heatmap data
  - Merge COBY heatmaps with core COB heatmaps
  - Provide unified heatmap interface to models
  - Support exchange-specific heatmap filtering
  - Cache merged heatmaps for performance
  - _Requirements: 1.4, 3.1_

- [ ] 3.3. Implement COBY health monitoring
  - Add COBY connection status to DataProvider
  - Monitor COBY API availability
  - Track COBY data freshness
  - Alert on COBY failures
  - Provide COBY status in dashboard
  - _Requirements: 1.6, 8.5_

### Phase 4: Model Output Management

- [ ] 4. Enhance ModelOutputManager functionality
  - Review model_output_manager.py implementation
  - Verify extensible ModelOutput format is working
  - Test cross-model feeding with hidden states
  - Validate historical output storage (1000 entries)
  - Optimize query performance by model_name, symbol, timestamp
  - _Requirements: 1.10, 8.2_

- [ ] 4.1. Implement model output persistence
  - Add disk-based storage for model outputs
  - Support configurable retention policies
  - Implement efficient serialization (pickle/msgpack)
  - Add compression for storage optimization
  - Support output replay for backtesting
  - _Requirements: 1.10, 5.7_

- [ ] 4.2. Create model output analytics
  - Track prediction accuracy over time
  - Calculate model agreement/disagreement metrics
  - Identify model performance patterns
  - Generate model comparison reports
  - Visualize model outputs in dashboard
  - _Requirements: 5.8, 10.7_

### Phase 5: Testing and Validation

- [ ] 5. Create comprehensive data provider tests
  - Write unit tests for DataProvider core functionality
  - Test automatic data maintenance worker
  - Test COB aggregation and imbalance calculations
  - Test Williams pivot point detection
  - Test StandardizedDataProvider validation
  - _Requirements: 8.1, 8.2_

- [ ] 5.1. Implement integration tests
  - Test end-to-end data flow from WebSocket to models
  - Test COBY integration (when implemented)
  - Test model output storage and retrieval
  - Test data provider under load
  - Test failure scenarios and recovery
  - _Requirements: 8.2, 8.3_

- [ ] 5.2. Create data provider performance benchmarks
  - Measure data collection latency
  - Measure COB aggregation performance
  - Measure BaseDataInput creation time
  - Identify performance bottlenecks
  - Optimize critical paths
  - _Requirements: 8.4_

- [ ] 5.3. Document data provider architecture
  - Create comprehensive architecture documentation
  - Document data flow diagrams
  - Document configuration options
  - Create troubleshooting guide
  - Add code examples for common use cases
  - _Requirements: 8.1, 8.2_

## Enhanced CNN Model Implementation

- [ ] 6. Enhance the existing CNN model with standardized inputs/outputs
  - Extend the current implementation in NN/models/enhanced_cnn.py
  - Accept standardized COB+OHLCV data frame: 300 frames (1s,1m,1h,1d) ETH + 300s 1s BTC
  - Include COB ±20 buckets and MA (1s,5s,15s,60s) of COB imbalance ±5 buckets
  - Output BUY/SELL trading action with confidence scores
  - _Requirements: 2.1, 2.2, 2.8, 1.10_

- [x] 6.1. Implement CNN inference with standardized input format
  - Accept BaseDataInput with standardized COB+OHLCV format
  - Process 300 frames of multi-timeframe data with COB buckets
  - Output BUY/SELL recommendations with confidence scores
  - Make hidden layer states available for cross-model feeding
  - Optimize inference performance for real-time processing
  - _Requirements: 2.2, 2.6, 2.8, 4.3_

- [x] 6.2. Enhance CNN training pipeline with checkpoint management
  - Integrate with checkpoint manager for training progress persistence
  - Store top 5-10 best checkpoints based on performance metrics
  - Automatically load best checkpoint at startup
  - Implement training triggers based on orchestrator feedback
  - Store metadata with checkpoints for performance tracking
  - _Requirements: 2.4, 2.5, 5.2, 5.3, 5.7_

- [ ] 6.3. Implement CNN model evaluation and checkpoint optimization
  - Create evaluation methods using standardized input/output format
  - Implement performance metrics for checkpoint ranking
  - Add validation against historical trading outcomes
  - Support automatic checkpoint cleanup (keep only top performers)
  - Track model improvement over time through checkpoint metadata
  - _Requirements: 2.5, 5.8, 4.4_

## Enhanced RL Model Implementation

- [ ] 7. Enhance the existing RL model with standardized inputs/outputs
  - Extend the current implementation in NN/models/dqn_agent.py
  - Accept standardized COB+OHLCV data frame: 300 frames (1s,1m,1h,1d) ETH + 300s 1s BTC
  - Include COB ±20 buckets and MA (1s,5s,15s,60s) of COB imbalance ±5 buckets
  - Output BUY/SELL trading action with confidence scores
  - _Requirements: 3.1, 3.2, 3.7, 1.10_

- [ ] 7.1. Implement RL inference with standardized input format
  - Accept BaseDataInput with standardized COB+OHLCV format
  - Process CNN hidden states and predictions as part of state input
  - Output BUY/SELL recommendations with confidence scores
  - Include expected rewards and value estimates in output
  - Optimize inference performance for real-time processing
  - _Requirements: 3.2, 3.7, 4.3_

- [ ] 7.2. Enhance RL training pipeline with checkpoint management
  - Integrate with checkpoint manager for training progress persistence
  - Store top 5-10 best checkpoints based on trading performance metrics
  - Automatically load best checkpoint at startup
  - Implement experience replay with profitability-based prioritization
  - Store metadata with checkpoints for performance tracking
  - _Requirements: 3.3, 3.5, 5.4, 5.7, 4.4_

- [ ] 7.3. Implement RL model evaluation and checkpoint optimization
  - Create evaluation methods using standardized input/output format
  - Implement trading performance metrics for checkpoint ranking
  - Add validation against historical trading opportunities
  - Support automatic checkpoint cleanup (keep only top performers)
  - Track model improvement over time through checkpoint metadata
  - _Requirements: 3.3, 5.8, 4.4_

## Enhanced Orchestrator Implementation

- [ ] 8. Enhance the existing orchestrator with centralized coordination
  - Extend the current implementation in core/orchestrator.py
  - Implement DataSubscriptionManager for multi-rate data streams
  - Add ModelInferenceCoordinator for cross-model coordination
  - Create ModelOutputStore for extensible model output management
  - Add TrainingPipelineManager for continuous learning coordination
  - _Requirements: 4.1, 4.2, 4.5, 8.1_

- [ ] 8.1. Implement data subscription and management system
  - Create DataSubscriptionManager class
  - Subscribe to 10Hz COB data, OHLCV, market ticks, and technical indicators
  - Implement intelligent caching for "last updated" data serving
  - Maintain synchronized base dataframe across different refresh rates
  - Add thread-safe access to multi-rate data streams
  - _Requirements: 4.1, 1.6, 8.5_

- [ ] 8.2. Implement model inference coordination
  - Create ModelInferenceCoordinator class
  - Trigger model inference based on data availability and requirements
  - Coordinate parallel inference execution for independent models
  - Handle model dependencies (e.g., RL waiting for CNN hidden states)
  - Assemble appropriate input data for each model type
  - _Requirements: 4.2, 3.1, 2.1_

- [ ] 8.3. Implement model output storage and cross-feeding
  - Create ModelOutputStore class using standardized ModelOutput format
  - Store CNN predictions, confidence scores, and hidden layer states
  - Store RL action recommendations and value estimates
  - Support extensible storage for LSTM, Transformer, and future models
  - Implement cross-model feeding of hidden states and predictions
  - Include "last predictions" from all models in base data input
  - _Requirements: 4.3, 1.10, 8.2_

- [ ] 8.4. Implement training pipeline management
  - Create TrainingPipelineManager class
  - Call each model's training pipeline with prediction-result pairs
  - Manage training data collection and labeling
  - Coordinate online learning updates based on real-time performance
  - Track prediction accuracy and trigger retraining when needed
  - _Requirements: 4.4, 5.2, 5.4, 5.7_

- [ ] 8.5. Implement enhanced decision-making with MoE
  - Create enhanced DecisionMaker class
  - Implement Mixture of Experts approach for model integration
  - Apply confidence-based filtering to avoid uncertain trades
  - Support configurable thresholds for buy/sell decisions
  - Consider market conditions and risk parameters in decisions
  - _Requirements: 4.5, 4.8, 6.7_

- [ ] 8.6. Implement extensible model integration architecture
  - Create MoEGateway class supporting dynamic model addition
  - Support CNN, RL, LSTM, Transformer model types without architecture changes
  - Implement model versioning and rollback capabilities
  - Handle model failures and fallback mechanisms
  - Provide model performance monitoring and alerting
  - _Requirements: 4.6, 8.2, 8.3_

## Model Inference Data Validation and Storage

- [x] 9. Implement comprehensive inference data validation system
  - Create InferenceDataValidator class for input validation
  - Validate complete OHLCV dataframes for all required timeframes
  - Check input data dimensions against model requirements
  - Log missing components and prevent prediction on incomplete data
  - _Requirements: 9.1, 9.2, 9.3, 9.4_

- [ ] 9.1. Implement input data validation for all models
  - Create validation methods for CNN, RL, and future model inputs
  - Validate OHLCV data completeness (300 frames for 1s, 1m, 1h, 1d)
  - Validate COB data structure (±20 buckets, MA calculations)
  - Raise specific validation errors with expected vs actual dimensions
  - Ensure validation occurs before any model inference
  - _Requirements: 9.1, 9.4_

- [x] 9.2. Implement persistent inference history storage
  - Create InferenceHistoryStore class for persistent storage
  - Store complete input data packages with each prediction
  - Include timestamp, symbol, input features, prediction outputs, confidence scores
  - Store model internal states for cross-model feeding
  - Implement compressed storage to minimize footprint
  - _Requirements: 9.5, 9.6_

- [x] 9.3. Implement inference history query and retrieval system
  - Create efficient query mechanisms by symbol, timeframe, and date range
  - Implement data retrieval for training pipeline consumption
  - Add data completeness metrics and validation results in storage
  - Handle storage failures gracefully without breaking prediction flow
  - _Requirements: 9.7, 11.6_

## Inference-Training Feedback Loop Implementation

- [ ] 10. Implement prediction outcome evaluation system
  - Create PredictionOutcomeEvaluator class
  - Evaluate prediction accuracy against actual price movements
  - Create training examples using stored inference data and actual outcomes
  - Feed prediction-result pairs back to respective models
  - _Requirements: 10.1, 10.2, 10.3_

- [ ] 10.1. Implement adaptive learning signal generation
  - Create positive reinforcement signals for accurate predictions
  - Generate corrective training signals for inaccurate predictions
  - Retrieve last inference data for each model for outcome comparison
  - Implement model-specific learning signal formats
  - _Requirements: 10.4, 10.5, 10.6_

- [ ] 10.2. Implement continuous improvement tracking
  - Track and report accuracy improvements/degradations over time
  - Monitor model learning progress through feedback loop
  - Create performance metrics for inference-training effectiveness
  - Generate alerts for learning regression or stagnation
  - _Requirements: 10.7_

## Inference History Management and Monitoring

- [ ] 11. Implement comprehensive inference logging and monitoring
  - Create InferenceMonitor class for logging and alerting
  - Log inference data storage operations with completeness metrics
  - Log training outcomes and model performance changes
  - Alert administrators on data flow issues with specific error details
  - _Requirements: 11.1, 11.2, 11.3_

- [ ] 11.1. Implement configurable retention policies
  - Create RetentionPolicyManager class
  - Archive or remove oldest entries when limits are reached
  - Prioritize keeping most recent and valuable training examples
  - Implement storage space monitoring and alerts
  - _Requirements: 11.4, 11.7_

- [ ] 11.2. Implement efficient historical data management
  - Compress inference data to minimize storage footprint
  - Maintain accessibility for training and analysis
  - Implement efficient query mechanisms for historical analysis
  - Add data archival and restoration capabilities
  - _Requirements: 11.5, 11.6_

## Trading Executor Implementation

- [ ] 12. Design and implement the trading executor
  - Create a TradingExecutor class that accepts trading actions from the orchestrator
  - Implement order execution through brokerage APIs
  - Add order lifecycle management
  - _Requirements: 7.1, 7.2, 8.6_

- [ ] 12.1. Implement brokerage API integrations
  - Create a BrokerageAPI interface
  - Implement concrete classes for MEXC and Binance
  - Add error handling and retry mechanisms
  - _Requirements: 7.1, 7.2, 8.6_

- [ ] 12.2. Implement order management
  - Create an OrderManager class
  - Implement methods for creating, updating, and canceling orders
  - Add order tracking and status updates
  - _Requirements: 7.1, 7.2, 8.6_

- [ ] 12.3. Implement error handling
  - Add comprehensive error handling for API failures
  - Implement circuit breakers for extreme market conditions
  - Add logging and notification mechanisms
  - _Requirements: 7.1, 7.2, 8.6_

## Risk Manager Implementation

- [ ] 13. Design and implement the risk manager
  - Create a RiskManager class
  - Implement risk parameter management
  - Add risk metric calculation
  - _Requirements: 7.1, 7.3, 7.4_

- [ ] 13.1. Implement stop-loss functionality
  - Create a StopLossManager class
  - Implement methods for creating and managing stop-loss orders
  - Add mechanisms to automatically close positions when stop-loss is triggered
  - _Requirements: 7.1, 7.2_

- [ ] 13.2. Implement position sizing
  - Create a PositionSizer class
  - Implement methods for calculating position sizes based on risk parameters
  - Add validation to ensure position sizes are within limits
  - _Requirements: 7.3, 7.7_

- [ ] 13.3. Implement risk metrics
  - Add methods to calculate risk metrics (drawdown, VaR, etc.)
  - Implement real-time risk monitoring
  - Add alerts for high-risk situations
  - _Requirements: 7.4, 7.5, 7.6, 7.8_

## Dashboard Implementation

- [ ] 14. Design and implement the dashboard UI
  - Create a Dashboard class
  - Implement the web-based UI using Flask/Dash
  - Add real-time updates using WebSockets
  - _Requirements: 6.1, 6.8_

- [ ] 14.1. Implement chart management
  - Create a ChartManager class
  - Implement methods for creating and updating charts
  - Add interactive features (zoom, pan, etc.)
  - _Requirements: 6.1, 6.2_

- [ ] 14.2. Implement control panel
  - Create a ControlPanel class
  - Implement start/stop toggles for system processes
  - Add sliders for adjusting buy/sell thresholds
  - _Requirements: 6.6, 6.7_

- [ ] 14.3. Implement system status display
  - Add methods to display training progress
  - Implement model performance metrics visualization
  - Add real-time system status updates
  - _Requirements: 6.5, 5.6_

- [ ] 14.4. Implement server-side processing
  - Ensure all processes run on the server without requiring the dashboard to be open
  - Implement background tasks for model training and inference
  - Add mechanisms to persist system state
  - _Requirements: 6.8, 5.5_

## Integration and Testing

- [ ] 15. Integrate all components
  - Connect the data provider to the CNN and RL models
  - Connect the CNN and RL models to the orchestrator
  - Connect the orchestrator to the trading executor
  - _Requirements: 8.1, 8.2, 8.3_

- [ ] 15.1. Implement comprehensive unit tests
  - Create unit tests for each component
  - Implement test fixtures and mocks
  - Add test coverage reporting
  - _Requirements: 8.1, 8.2, 8.3_

- [ ] 15.2. Implement integration tests
  - Create tests for component interactions
  - Implement end-to-end tests
  - Add performance benchmarks
  - _Requirements: 8.1, 8.2, 8.3_

- [ ] 15.3. Implement backtesting framework
  - Create a backtesting environment
  - Implement methods to replay historical data
  - Add performance metrics calculation
  - _Requirements: 5.8, 8.1_

- [ ] 15.4. Optimize performance
  - Profile the system to identify bottlenecks
  - Implement optimizations for critical paths
  - Add caching and parallelization where appropriate
  - _Requirements: 8.1, 8.2, 8.3_