popov/gogo2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

more cleanup

Browse Source
This commit is contained in:
Dobromir Popov
2025-10-13 16:11:06 +03:00
parent 6cf4d902df
commit 0c28a0997c
17 changed files with 1030 additions and 2301 deletions
Download Patch File Download Diff File Expand all files Collapse all files

324
.kiro/specs/1.multi-modal-trading-system/tasks.md
View File

@@ -1,67 +1,206 @@
# Implementation Plan
## Enhanced Data Provider and COB Integration
## Data Provider Backbone Enhancement
- [ ] 1. Enhance the existing DataProvider class with standardized model inputs
- Extend the current implementation in core/data_provider.py
- Implement standardized COB+OHLCV data frame for all models
- Create unified input format: 300 frames OHLCV (1s, 1m, 1h, 1d) ETH + 300s of 1s BTC
- Integrate with existing multi_exchange_cob_provider.py for COB data
- _Requirements: 1.1, 1.2, 1.3, 1.6_
### Phase 1: Core Data Provider Enhancements
- [ ] 1.1. Implement standardized COB+OHLCV data frame for all models
- Create BaseDataInput class with standardized format for all models
- Implement OHLCV: 300 frames of (1s, 1m, 1h, 1d) ETH + 300s of 1s BTC
- Add COB: ±20 buckets of COB amounts in USD for each 1s OHLCV
- Include 1s, 5s, 15s, and 60s MA of COB imbalance counting ±5 COB buckets
- Ensure all models receive identical input format for consistency
- _Requirements: 1.2, 1.3, 8.1_
- [ ] 1. Audit and validate existing DataProvider implementation
- Review core/data_provider.py for completeness and correctness
- Validate 1500-candle caching is working correctly
- Verify automatic data maintenance worker is updating properly
- Test fallback mechanisms between Binance and MEXC
- Document any gaps or issues found
- _Requirements: 1.1, 1.2, 1.6_
- [ ] 1.2. Implement extensible model output storage
- Create standardized ModelOutput data structure
- Support CNN, RL, LSTM, Transformer, and future model types
- Include model-specific predictions and cross-model hidden states
- Add metadata support for extensible model information
- _Requirements: 1.10, 8.2_
- [ ] 1.1. Enhance COB data collection robustness
- Fix 'NoneType' object has no attribute 'append' errors in _cob_aggregation_worker
- Add defensive checks before accessing deque structures
- Implement proper initialization guards to prevent duplicate COB collection starts
- Add comprehensive error logging for COB data processing failures
- Test COB collection under various failure scenarios
- _Requirements: 1.3, 1.6_
- [ ] 1.3. Enhance Williams Market Structure pivot point calculation
- Extend existing williams_market_structure.py implementation
- Improve recursive pivot point calculation accuracy
- Add unit tests to verify pivot point detection
- Integrate with COB data for enhanced pivot detection
- [ ] 1.2. Implement configurable COB price ranges
- Replace hardcoded price ranges ($5 ETH, $50 BTC) with configuration
- Add _get_price_range_for_symbol() configuration support
- Allow per-symbol price range customization via config.yaml
- Update COB imbalance calculations to use configurable ranges
- Document price range selection rationale
- _Requirements: 1.4, 1.1_
- [ ] 1.3. Validate and enhance Williams Market Structure pivot calculation
- Review williams_market_structure.py implementation
- Verify 5-level pivot detection is working correctly
- Test monthly 1s data analysis for comprehensive context
- Add unit tests for pivot point detection accuracy
- Optimize pivot calculation performance if needed
- _Requirements: 1.5, 2.7_
- [x] 1.4. Optimize real-time data streaming with COB integration
- [ ] 1.4. Implement COB heatmap matrix generation
- Create get_cob_heatmap_matrix() method in DataProvider
- Generate time x price matrix for visualization and model input
- Support configurable time windows (default 300 seconds)
- Support configurable price bucket radius (default ±10 buckets)
- Support multiple metrics (imbalance, volume, spread)
- Cache heatmap data for performance
- _Requirements: 1.4, 1.1_
- Enhance existing WebSocket connections in enhanced_cob_websocket.py
- Implement 10Hz COB data streaming alongside OHLCV data
- Add data synchronization across different refresh rates
- Ensure thread-safe access to multi-rate data streams
- [x] 1.5. Enhance EnhancedCOBWebSocket reliability
- Review enhanced_cob_websocket.py for stability issues
- Verify proper order book synchronization with REST snapshots
- Test reconnection logic with exponential backoff
- Ensure 24-hour connection limit compliance
- Add comprehensive error handling for all WebSocket streams
- _Requirements: 1.3, 1.6_
### Phase 2: StandardizedDataProvider Enhancements
- [ ] 2. Implement comprehensive BaseDataInput validation
- Enhance validate() method in BaseDataInput dataclass
- Add minimum frame count validation (100 frames per timeframe)
- Implement data completeness scoring (0.0 to 1.0)
- Add COB data validation (non-null, valid buckets)
- Create detailed validation error messages
- Prevent model inference on incomplete data (completeness < 0.8)
- _Requirements: 1.1.2, 1.1.6_
- [ ] 2.1. Integrate COB heatmap into BaseDataInput
- Add cob_heatmap_times, cob_heatmap_prices, cob_heatmap_values fields
- Call get_cob_heatmap_matrix() in get_base_data_input()
- Handle heatmap generation failures gracefully
- Store heatmap mid_prices in market_microstructure
- Document heatmap usage for models
- _Requirements: 1.1.1, 1.4_
- [ ] 2.2. Enhance COB moving average calculation
- Review _calculate_cob_moving_averages() for correctness
- Fix bucket quantization to match COB snapshot buckets
- Implement nearest-key matching for historical imbalance lookup
- Add thread-safe access to cob_imbalance_history
- Optimize MA calculation performance
- _Requirements: 1.1.3, 1.4_
- [ ] 2.3. Implement data quality scoring system
- Create data_quality_score() method
- Score based on: data completeness, freshness, consistency
- Add quality thresholds for model inference
- Log quality metrics for monitoring
- Provide quality breakdown in BaseDataInput
- _Requirements: 1.1.2, 1.1.6_
- [ ] 2.4. Enhance live price fetching robustness
- Review get_live_price_from_api() fallback chain
- Add retry logic with exponential backoff
- Implement circuit breaker for repeated API failures
- Cache prices with configurable TTL (default 500ms)
- Log price source for debugging
- _Requirements: 1.6, 1.7_
### Phase 3: COBY Integration
- [ ] 3. Design unified interface between COBY and core DataProvider
- Define clear boundaries between COBY and core systems
- Create adapter layer for accessing COBY data from core
- Design data flow for multi-exchange aggregation
- Plan migration path for existing code
- Document integration architecture
- _Requirements: 1.10, 8.1_
- [ ] 3.1. Implement COBY data access adapter
- Create COBYDataAdapter class in core/
- Implement methods to query COBY TimescaleDB
- Add Redis cache integration for performance
- Support historical data retrieval from COBY
- Handle COBY unavailability gracefully
- _Requirements: 1.10, 8.1_
- [ ] 3.2. Integrate COBY heatmap data
- Query COBY for multi-exchange heatmap data
- Merge COBY heatmaps with core COB heatmaps
- Provide unified heatmap interface to models
- Support exchange-specific heatmap filtering
- Cache merged heatmaps for performance
- _Requirements: 1.4, 3.1_
- [ ] 3.3. Implement COBY health monitoring
- Add COBY connection status to DataProvider
- Monitor COBY API availability
- Track COBY data freshness
- Alert on COBY failures
- Provide COBY status in dashboard
- _Requirements: 1.6, 8.5_
- [ ] 1.5. Fix WebSocket COB data processing errors
- Fix 'NoneType' object has no attribute 'append' errors in COB data processing
- Ensure proper initialization of data structures in MultiExchangeCOBProvider
- Add validation and defensive checks before accessing data structures
- Implement proper error handling for WebSocket data processing
- _Requirements: 1.1, 1.6, 8.5_
### Phase 4: Model Output Management
- [ ] 1.6. Enhance error handling in COB data processing
- Add validation for incoming WebSocket data
- Implement reconnection logic with exponential backoff
- Add detailed logging for debugging COB data issues
- Ensure system continues operation with last valid data during failures
- _Requirements: 1.6, 8.5_
- [ ] 4. Enhance ModelOutputManager functionality
- Review model_output_manager.py implementation
- Verify extensible ModelOutput format is working
- Test cross-model feeding with hidden states
- Validate historical output storage (1000 entries)
- Optimize query performance by model_name, symbol, timestamp
- _Requirements: 1.10, 8.2_
- [ ] 4.1. Implement model output persistence
- Add disk-based storage for model outputs
- Support configurable retention policies
- Implement efficient serialization (pickle/msgpack)
- Add compression for storage optimization
- Support output replay for backtesting
- _Requirements: 1.10, 5.7_
- [ ] 4.2. Create model output analytics
- Track prediction accuracy over time
- Calculate model agreement/disagreement metrics
- Identify model performance patterns
- Generate model comparison reports
- Visualize model outputs in dashboard
- _Requirements: 5.8, 10.7_
### Phase 5: Testing and Validation
- [ ] 5. Create comprehensive data provider tests
- Write unit tests for DataProvider core functionality
- Test automatic data maintenance worker
- Test COB aggregation and imbalance calculations
- Test Williams pivot point detection
- Test StandardizedDataProvider validation
- _Requirements: 8.1, 8.2_
- [ ] 5.1. Implement integration tests
- Test end-to-end data flow from WebSocket to models
- Test COBY integration (when implemented)
- Test model output storage and retrieval
- Test data provider under load
- Test failure scenarios and recovery
- _Requirements: 8.2, 8.3_
- [ ] 5.2. Create data provider performance benchmarks
- Measure data collection latency
- Measure COB aggregation performance
- Measure BaseDataInput creation time
- Identify performance bottlenecks
- Optimize critical paths
- _Requirements: 8.4_
- [ ] 5.3. Document data provider architecture
- Create comprehensive architecture documentation
- Document data flow diagrams
- Document configuration options
- Create troubleshooting guide
- Add code examples for common use cases
- _Requirements: 8.1, 8.2_
## Enhanced CNN Model Implementation
- [ ] 2. Enhance the existing CNN model with standardized inputs/outputs
- [ ] 6. Enhance the existing CNN model with standardized inputs/outputs
- Extend the current implementation in NN/models/enhanced_cnn.py
- Accept standardized COB+OHLCV data frame: 300 frames (1s,1m,1h,1d) ETH + 300s 1s BTC
- Include COB ±20 buckets and MA (1s,5s,15s,60s) of COB imbalance ±5 buckets
- Output BUY/SELL trading action with confidence scores - _Requirements: 2.1, 2.2, 2.8, 1.10_
- Output BUY/SELL trading action with confidence scores
- _Requirements: 2.1, 2.2, 2.8, 1.10_
- [x] 2.1. Implement CNN inference with standardized input format
- [x] 6.1. Implement CNN inference with standardized input format
- Accept BaseDataInput with standardized COB+OHLCV format
- Process 300 frames of multi-timeframe data with COB buckets
- Output BUY/SELL recommendations with confidence scores
@@ -69,7 +208,7 @@
- Optimize inference performance for real-time processing
- _Requirements: 2.2, 2.6, 2.8, 4.3_
- [x] 2.2. Enhance CNN training pipeline with checkpoint management
- [x] 6.2. Enhance CNN training pipeline with checkpoint management
- Integrate with checkpoint manager for training progress persistence
- Store top 5-10 best checkpoints based on performance metrics
- Automatically load best checkpoint at startup
@@ -77,7 +216,7 @@
- Store metadata with checkpoints for performance tracking
- _Requirements: 2.4, 2.5, 5.2, 5.3, 5.7_
- [ ] 2.3. Implement CNN model evaluation and checkpoint optimization
- [ ] 6.3. Implement CNN model evaluation and checkpoint optimization
- Create evaluation methods using standardized input/output format
- Implement performance metrics for checkpoint ranking
- Add validation against historical trading outcomes
@@ -87,14 +226,14 @@
## Enhanced RL Model Implementation
- [ ] 3. Enhance the existing RL model with standardized inputs/outputs
- [ ] 7. Enhance the existing RL model with standardized inputs/outputs
- Extend the current implementation in NN/models/dqn_agent.py
- Accept standardized COB+OHLCV data frame: 300 frames (1s,1m,1h,1d) ETH + 300s 1s BTC
- Include COB ±20 buckets and MA (1s,5s,15s,60s) of COB imbalance ±5 buckets
- Output BUY/SELL trading action with confidence scores
- _Requirements: 3.1, 3.2, 3.7, 1.10_
- [ ] 3.1. Implement RL inference with standardized input format
- [ ] 7.1. Implement RL inference with standardized input format
- Accept BaseDataInput with standardized COB+OHLCV format
- Process CNN hidden states and predictions as part of state input
- Output BUY/SELL recommendations with confidence scores
@@ -102,7 +241,7 @@
- Optimize inference performance for real-time processing
- _Requirements: 3.2, 3.7, 4.3_
- [ ] 3.2. Enhance RL training pipeline with checkpoint management
- [ ] 7.2. Enhance RL training pipeline with checkpoint management
- Integrate with checkpoint manager for training progress persistence
- Store top 5-10 best checkpoints based on trading performance metrics
- Automatically load best checkpoint at startup
@@ -110,7 +249,7 @@
- Store metadata with checkpoints for performance tracking
- _Requirements: 3.3, 3.5, 5.4, 5.7, 4.4_
- [ ] 3.3. Implement RL model evaluation and checkpoint optimization
- [ ] 7.3. Implement RL model evaluation and checkpoint optimization
- Create evaluation methods using standardized input/output format
- Implement trading performance metrics for checkpoint ranking
- Add validation against historical trading opportunities
@@ -120,7 +259,7 @@
## Enhanced Orchestrator Implementation
- [ ] 4. Enhance the existing orchestrator with centralized coordination
- [ ] 8. Enhance the existing orchestrator with centralized coordination
- Extend the current implementation in core/orchestrator.py
- Implement DataSubscriptionManager for multi-rate data streams
- Add ModelInferenceCoordinator for cross-model coordination
@@ -128,7 +267,7 @@
- Add TrainingPipelineManager for continuous learning coordination
- _Requirements: 4.1, 4.2, 4.5, 8.1_
- [ ] 4.1. Implement data subscription and management system
- [ ] 8.1. Implement data subscription and management system
- Create DataSubscriptionManager class
- Subscribe to 10Hz COB data, OHLCV, market ticks, and technical indicators
- Implement intelligent caching for "last updated" data serving
@@ -136,10 +275,7 @@
- Add thread-safe access to multi-rate data streams
- _Requirements: 4.1, 1.6, 8.5_
- [ ] 4.2. Implement model inference coordination
- [ ] 8.2. Implement model inference coordination
- Create ModelInferenceCoordinator class
- Trigger model inference based on data availability and requirements
- Coordinate parallel inference execution for independent models
@@ -147,7 +283,7 @@
- Assemble appropriate input data for each model type
- _Requirements: 4.2, 3.1, 2.1_
- [ ] 4.3. Implement model output storage and cross-feeding
- [ ] 8.3. Implement model output storage and cross-feeding
- Create ModelOutputStore class using standardized ModelOutput format
- Store CNN predictions, confidence scores, and hidden layer states
- Store RL action recommendations and value estimates
@@ -156,7 +292,7 @@
- Include "last predictions" from all models in base data input
- _Requirements: 4.3, 1.10, 8.2_
- [ ] 4.4. Implement training pipeline management
- [ ] 8.4. Implement training pipeline management
- Create TrainingPipelineManager class
- Call each model's training pipeline with prediction-result pairs
- Manage training data collection and labeling
@@ -164,7 +300,7 @@
- Track prediction accuracy and trigger retraining when needed
- _Requirements: 4.4, 5.2, 5.4, 5.7_
- [ ] 4.5. Implement enhanced decision-making with MoE
- [ ] 8.5. Implement enhanced decision-making with MoE
- Create enhanced DecisionMaker class
- Implement Mixture of Experts approach for model integration
- Apply confidence-based filtering to avoid uncertain trades
@@ -172,7 +308,7 @@
- Consider market conditions and risk parameters in decisions
- _Requirements: 4.5, 4.8, 6.7_
- [ ] 4.6. Implement extensible model integration architecture
- [ ] 8.6. Implement extensible model integration architecture
- Create MoEGateway class supporting dynamic model addition
- Support CNN, RL, LSTM, Transformer model types without architecture changes
- Implement model versioning and rollback capabilities
@@ -182,15 +318,14 @@
## Model Inference Data Validation and Storage
- [x] 5. Implement comprehensive inference data validation system
- [x] 9. Implement comprehensive inference data validation system
- Create InferenceDataValidator class for input validation
- Validate complete OHLCV dataframes for all required timeframes
- Check input data dimensions against model requirements
- Log missing components and prevent prediction on incomplete data
- _Requirements: 9.1, 9.2, 9.3, 9.4_
- [ ] 5.1. Implement input data validation for all models
- [ ] 9.1. Implement input data validation for all models
- Create validation methods for CNN, RL, and future model inputs
- Validate OHLCV data completeness (300 frames for 1s, 1m, 1h, 1d)
- Validate COB data structure (±20 buckets, MA calculations)
@@ -198,9 +333,7 @@
- Ensure validation occurs before any model inference
- _Requirements: 9.1, 9.4_
- [x] 5.2. Implement persistent inference history storage
- [x] 9.2. Implement persistent inference history storage
- Create InferenceHistoryStore class for persistent storage
- Store complete input data packages with each prediction
- Include timestamp, symbol, input features, prediction outputs, confidence scores
@@ -208,12 +341,7 @@
- Implement compressed storage to minimize footprint
- _Requirements: 9.5, 9.6_
- [x] 5.3. Implement inference history query and retrieval system
- [x] 9.3. Implement inference history query and retrieval system
- Create efficient query mechanisms by symbol, timeframe, and date range
- Implement data retrieval for training pipeline consumption
- Add data completeness metrics and validation results in storage
@@ -222,21 +350,21 @@
## Inference-Training Feedback Loop Implementation
- [ ] 6. Implement prediction outcome evaluation system
- [ ] 10. Implement prediction outcome evaluation system
- Create PredictionOutcomeEvaluator class
- Evaluate prediction accuracy against actual price movements
- Create training examples using stored inference data and actual outcomes
- Feed prediction-result pairs back to respective models
- _Requirements: 10.1, 10.2, 10.3_
- [ ] 6.1. Implement adaptive learning signal generation
- [ ] 10.1. Implement adaptive learning signal generation
- Create positive reinforcement signals for accurate predictions
- Generate corrective training signals for inaccurate predictions
- Retrieve last inference data for each model for outcome comparison
- Implement model-specific learning signal formats
- _Requirements: 10.4, 10.5, 10.6_
- [ ] 6.2. Implement continuous improvement tracking
- [ ] 10.2. Implement continuous improvement tracking
- Track and report accuracy improvements/degradations over time
- Monitor model learning progress through feedback loop
- Create performance metrics for inference-training effectiveness
@@ -245,21 +373,21 @@
## Inference History Management and Monitoring
- [ ] 7. Implement comprehensive inference logging and monitoring
- [ ] 11. Implement comprehensive inference logging and monitoring
- Create InferenceMonitor class for logging and alerting
- Log inference data storage operations with completeness metrics
- Log training outcomes and model performance changes
- Alert administrators on data flow issues with specific error details
- _Requirements: 11.1, 11.2, 11.3_
- [ ] 7.1. Implement configurable retention policies
- [ ] 11.1. Implement configurable retention policies
- Create RetentionPolicyManager class
- Archive or remove oldest entries when limits are reached
- Prioritize keeping most recent and valuable training examples
- Implement storage space monitoring and alerts
- _Requirements: 11.4, 11.7_
- [ ] 7.2. Implement efficient historical data management
- [ ] 11.2. Implement efficient historical data management
- Compress inference data to minimize storage footprint
- Maintain accessibility for training and analysis
- Implement efficient query mechanisms for historical analysis
@@ -268,25 +396,25 @@
## Trading Executor Implementation
- [ ] 5. Design and implement the trading executor
- [ ] 12. Design and implement the trading executor
- Create a TradingExecutor class that accepts trading actions from the orchestrator
- Implement order execution through brokerage APIs
- Add order lifecycle management
- _Requirements: 7.1, 7.2, 8.6_
- [ ] 5.1. Implement brokerage API integrations
- [ ] 12.1. Implement brokerage API integrations
- Create a BrokerageAPI interface
- Implement concrete classes for MEXC and Binance
- Add error handling and retry mechanisms
- _Requirements: 7.1, 7.2, 8.6_
- [ ] 5.2. Implement order management
- [ ] 12.2. Implement order management
- Create an OrderManager class
- Implement methods for creating, updating, and canceling orders
- Add order tracking and status updates
- _Requirements: 7.1, 7.2, 8.6_
- [ ] 5.3. Implement error handling
- [ ] 12.3. Implement error handling
- Add comprehensive error handling for API failures
- Implement circuit breakers for extreme market conditions
- Add logging and notification mechanisms
@@ -294,25 +422,25 @@
## Risk Manager Implementation
- [ ] 6. Design and implement the risk manager
- [ ] 13. Design and implement the risk manager
- Create a RiskManager class
- Implement risk parameter management
- Add risk metric calculation
- _Requirements: 7.1, 7.3, 7.4_
- [ ] 6.1. Implement stop-loss functionality
- [ ] 13.1. Implement stop-loss functionality
- Create a StopLossManager class
- Implement methods for creating and managing stop-loss orders
- Add mechanisms to automatically close positions when stop-loss is triggered
- _Requirements: 7.1, 7.2_
- [ ] 6.2. Implement position sizing
- [ ] 13.2. Implement position sizing
- Create a PositionSizer class
- Implement methods for calculating position sizes based on risk parameters
- Add validation to ensure position sizes are within limits
- _Requirements: 7.3, 7.7_
- [ ] 6.3. Implement risk metrics
- [ ] 13.3. Implement risk metrics
- Add methods to calculate risk metrics (drawdown, VaR, etc.)
- Implement real-time risk monitoring
- Add alerts for high-risk situations
@@ -320,31 +448,31 @@
## Dashboard Implementation
- [ ] 7. Design and implement the dashboard UI
- [ ] 14. Design and implement the dashboard UI
- Create a Dashboard class
- Implement the web-based UI using Flask/Dash
- Add real-time updates using WebSockets
- _Requirements: 6.1, 6.8_
- [ ] 7.1. Implement chart management
- [ ] 14.1. Implement chart management
- Create a ChartManager class
- Implement methods for creating and updating charts
- Add interactive features (zoom, pan, etc.)
- _Requirements: 6.1, 6.2_
- [ ] 7.2. Implement control panel
- [ ] 14.2. Implement control panel
- Create a ControlPanel class
- Implement start/stop toggles for system processes
- Add sliders for adjusting buy/sell thresholds
- _Requirements: 6.6, 6.7_
- [ ] 7.3. Implement system status display
- [ ] 14.3. Implement system status display
- Add methods to display training progress
- Implement model performance metrics visualization
- Add real-time system status updates
- _Requirements: 6.5, 5.6_
- [ ] 7.4. Implement server-side processing
- [ ] 14.4. Implement server-side processing
- Ensure all processes run on the server without requiring the dashboard to be open
- Implement background tasks for model training and inference
- Add mechanisms to persist system state
@@ -352,32 +480,32 @@
## Integration and Testing
- [ ] 8. Integrate all components
- [ ] 15. Integrate all components
- Connect the data provider to the CNN and RL models
- Connect the CNN and RL models to the orchestrator
- Connect the orchestrator to the trading executor
- _Requirements: 8.1, 8.2, 8.3_
- [ ] 8.1. Implement comprehensive unit tests
- [ ] 15.1. Implement comprehensive unit tests
- Create unit tests for each component
- Implement test fixtures and mocks
- Add test coverage reporting
- _Requirements: 8.1, 8.2, 8.3_
- [ ] 8.2. Implement integration tests
- [ ] 15.2. Implement integration tests
- Create tests for component interactions
- Implement end-to-end tests
- Add performance benchmarks
- _Requirements: 8.1, 8.2, 8.3_
- [ ] 8.3. Implement backtesting framework
- [ ] 15.3. Implement backtesting framework
- Create a backtesting environment
- Implement methods to replay historical data
- Add performance metrics calculation
- _Requirements: 5.8, 8.1_
- [ ] 8.4. Optimize performance
- [ ] 15.4. Optimize performance
- Profile the system to identify bottlenecks
- Implement optimizations for critical paths
- Add caching and parallelization where appropriate
- _Requirements: 8.1, 8.2, 8.3_
- _Requirements: 8.1, 8.2, 8.3_