popov/gogo2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

cleanup

Browse Source
This commit is contained in:
Dobromir Popov
2025-07-27 18:31:30 +03:00
parent a94b80c1f4
commit e2c495d83c
15 changed files with 3883 additions and 3335 deletions
Download Patch File Download Diff File Expand all files Collapse all files

125
COB_DATA_IMPROVEMENTS_SUMMARY.md Normal file
View File

@ -0,0 +1,125 @@
# COB Data Improvements Summary
## ✅ **Completed Improvements**
### 1. Fixed DateTime Comparison Error
- **Issue**: `'<=' not supported between instances of 'datetime.datetime' and 'float'`
- **Fix**: Added proper timestamp handling in `_aggregate_cob_1s()` method
- **Result**: COB aggregation now works without datetime errors
### 2. Added Multi-timeframe Imbalance Indicators
- **Added Indicators**:
- `imbalance_1s`: Current 1-second imbalance
- `imbalance_5s`: 5-second weighted average imbalance
- `imbalance_15s`: 15-second weighted average imbalance
- `imbalance_60s`: 60-second weighted average imbalance
- **Calculation Method**: Volume-weighted average with fallback to simple average
- **Storage**: Added to both main data structure and stats section
### 3. Enhanced COB Data Structure
- **Price Bucketing**: $1 USD price buckets for better granularity
- **Volume Tracking**: Separate bid/ask volume tracking
- **Statistics**: Comprehensive stats including spread, mid-price, volume
- **Imbalance Calculation**: Proper bid-ask imbalance: `(bid_vol - ask_vol) / total_vol`
### 4. Added COB Data Quality Monitoring
- **New Method**: `get_cob_data_quality()`
- **Metrics Tracked**:
- Raw tick count and freshness
- Aggregated data count and freshness
- Latest imbalance indicators
- Data freshness assessment (excellent/good/fair/stale/no_data)
- Price bucket counts
### 5. Improved Error Handling
- **Robust Timestamp Handling**: Supports both datetime and float timestamps
- **Graceful Degradation**: Returns default values when calculations fail
- **Comprehensive Logging**: Detailed error messages for debugging
## 📊 **Test Results**
### Mock Data Test Results:
- **✅ COB Aggregation**: Successfully processes ticks and creates 1s aggregated data
- **✅ Imbalance Calculation**:
- 1s imbalance: 0.1044 (from current tick)
- Multi-timeframe: 0.0000 (needs more historical data)
- **✅ Price Bucketing**: 6 buckets created (3 bid + 3 ask)
- **✅ Volume Tracking**: 594.00 total volume calculated correctly
- **✅ Quality Monitoring**: All metrics properly reported
### Real-time Data Status:
- **⚠️ WebSocket Connection**: Connecting but not receiving data yet
- **❌ COB Provider Error**: `MultiExchangeCOBProvider.__init__() got an unexpected keyword argument 'bucket_size_bps'`
- **✅ Data Structure**: Ready to receive and process real COB data
## 🔧 **Current Issues**
### 1. COB Provider Initialization Error
- **Error**: `bucket_size_bps` parameter not recognized
- **Impact**: Real COB data not flowing through system
- **Status**: Needs investigation of COB provider interface
### 2. WebSocket Data Flow
- **Status**: WebSocket connects but no data received yet
- **Possible Causes**:
- COB provider initialization failure
- WebSocket callback not properly connected
- Data format mismatch
## 📈 **Data Quality Indicators**
### Imbalance Indicators (Working):
```python
{
'imbalance_1s': 0.1044, # Current 1s imbalance
'imbalance_5s': 0.0000, # 5s weighted average
'imbalance_15s': 0.0000, # 15s weighted average
'imbalance_60s': 0.0000, # 60s weighted average
'total_volume': 594.00, # Total volume
'bucket_count': 6 # Price buckets
}
```
### Data Freshness Assessment:
- **excellent**: Data < 5 seconds old
- **good**: Data < 15 seconds old
- **fair**: Data < 60 seconds old
- **stale**: Data > 60 seconds old
- **no_data**: No data available
## 🎯 **Next Steps**
### 1. Fix COB Provider Integration
- Investigate `bucket_size_bps` parameter issue
- Ensure proper COB provider initialization
- Test real WebSocket data flow
### 2. Validate Real-time Imbalances
- Test with live market data
- Verify multi-timeframe calculations
- Monitor data quality in production
### 3. Integration Testing
- Test with trading models
- Verify dashboard integration
- Performance testing under load
## 🔍 **Usage Examples**
### Get COB Data Quality:
```python
dp = DataProvider()
quality = dp.get_cob_data_quality()
print(f"ETH imbalance 1s: {quality['imbalance_indicators']['ETH/USDT']['imbalance_1s']}")
```
### Get Recent Aggregated Data:
```python
recent_cob = dp.get_cob_1s_aggregated('ETH/USDT', count=10)
for record in recent_cob:
print(f"Time: {record['timestamp']}, Imbalance: {record['imbalance_1s']:.4f}")
```
## ✅ **Summary**
The COB data improvements are **functionally complete** and **tested**. The imbalance calculation system works correctly with multi-timeframe indicators. The main remaining issue is the COB provider initialization error that prevents real-time data flow. Once this is resolved, the system will provide high-quality COB data with comprehensive imbalance indicators for trading models.

112
DATA_PROVIDER_CHANGES_SUMMARY.md Normal file
View File

@ -0,0 +1,112 @@
# Data Provider Simplification Summary
## Changes Made
### 1. Removed Pre-loading System
- Removed `_should_preload_data()` method
- Removed `_preload_300s_data()` method
- Removed `preload_all_symbols_data()` method
- Removed all pre-loading logic from `get_historical_data()`
### 2. Simplified Data Structure
- Fixed symbols to `['ETH/USDT', 'BTC/USDT']`
- Fixed timeframes to `['1s', '1m', '1h', '1d']`
- Replaced `historical_data` with `cached_data` structure
- Each symbol/timeframe maintains exactly 1500 OHLCV candles (limited by API to ~1000)
### 3. Automatic Data Maintenance System
- Added `start_automatic_data_maintenance()` method
- Added `_data_maintenance_worker()` background thread
- Added `_initial_data_load()` for startup data loading
- Added `_update_cached_data()` for periodic updates
### 4. Data Update Strategy
- Initial load: Fetch 1500 candles for each symbol/timeframe at startup
- Periodic updates: Fetch last 2 candles every half candle period
- 1s data: Update every 0.5 seconds
- 1m data: Update every 30 seconds
- 1h data: Update every 30 minutes
- 1d data: Update every 12 hours
### 5. API Call Isolation
- `get_historical_data()` now only returns cached data
- No external API calls triggered by data requests
- All API calls happen in background maintenance thread
- Rate limiting increased to 500ms between requests
### 6. Updated Methods
- `get_historical_data()`: Returns cached data only
- `get_latest_candles()`: Uses cached data + real-time data
- `get_current_price()`: Uses cached data only
- `get_price_at_index()`: Uses cached data only
- `get_feature_matrix()`: Uses cached data only
- `_get_cached_ohlcv_bars()`: Simplified to use cached data
- `health_check()`: Updated to show cached data status
### 7. New Methods Added
- `get_cached_data_summary()`: Returns detailed cache status
- `stop_automatic_data_maintenance()`: Stops background updates
### 8. Removed Methods
- All pre-loading related methods
- `invalidate_ohlcv_cache()` (no longer needed)
- `_build_ohlcv_bar_cache()` (simplified)
## Test Results
### ✅ **Test Script Results:**
- **Initial Data Load**: Successfully loaded 1000 candles for each symbol/timeframe
- **Cached Data Access**: `get_historical_data()` returns cached data without API calls
- **Current Price Retrieval**: Works correctly from cached data (ETH: $3,809, BTC: $118,290)
- **Automatic Updates**: Background maintenance thread updating data every half candle period
- **WebSocket Integration**: COB WebSocket connecting and working properly
### 📊 **Data Loaded:**
- **ETH/USDT**: 1s, 1m, 1h, 1d (1000 candles each)
- **BTC/USDT**: 1s, 1m, 1h, 1d (1000 candles each)
- **Total**: 8,000 OHLCV candles cached and maintained automatically
### 🔧 **Minor Issues:**
- Initial load gets ~1000 candles instead of 1500 (Binance API limit)
- Some WebSocket warnings on Windows (non-critical)
- COB provider initialization error (doesn't affect main functionality)
## Benefits
1. **Predictable Performance**: No unexpected API calls during data requests
2. **Rate Limit Compliance**: All API calls controlled in background thread
3. **Consistent Data**: Always 1000+ candles available for each symbol/timeframe
4. **Real-time Updates**: Data stays fresh with automatic background updates
5. **Simplified Architecture**: Clear separation between data access and data fetching
## Usage
```python
# Initialize data provider (starts automatic maintenance)
dp = DataProvider()
# Get cached data (no API calls)
data = dp.get_historical_data('ETH/USDT', '1m', limit=100)
# Get current price from cache
price = dp.get_current_price('ETH/USDT')
# Check cache status
summary = dp.get_cached_data_summary()
# Stop maintenance when done
dp.stop_automatic_data_maintenance()
```
## Test Scripts
- `test_simplified_data_provider.py`: Basic functionality test
- `example_usage_simplified_data_provider.py`: Comprehensive usage examples
## Performance Metrics
- **Startup Time**: ~15 seconds for initial data load
- **Memory Usage**: ~8,000 OHLCV candles in memory
- **API Calls**: Controlled background updates only
- **Data Freshness**: Updated every half candle period
- **Cache Hit Rate**: 100% for data requests (no API calls)

276
core/cnn_dashboard_integration.py
View File

@ -1,276 +0,0 @@
"""
CNN Dashboard Integration
This module integrates the EnhancedCNN model with the dashboard, providing real-time
training and visualization of model predictions.
"""
import logging
import threading
import time
from datetime import datetime
from typing import Dict, List, Optional, Any, Tuple
import os
import json
from .enhanced_cnn_adapter import EnhancedCNNAdapter
from .data_models import BaseDataInput, ModelOutput, create_model_output
from utils.training_integration import get_training_integration
logger = logging.getLogger(__name__)
class CNNDashboardIntegration:
"""
Integrates the EnhancedCNN model with the dashboard
This class:
1. Loads and initializes the CNN model
2. Processes real-time data for model inference
3. Manages continuous training of the model
4. Provides visualization data for the dashboard
"""
def __init__(self, data_provider=None, checkpoint_dir: str = "models/enhanced_cnn"):
"""
Initialize the CNN dashboard integration
Args:
data_provider: Data provider instance
checkpoint_dir: Directory to save checkpoints to
"""
self.data_provider = data_provider
self.checkpoint_dir = checkpoint_dir
self.cnn_adapter = None
self.training_thread = None
self.training_active = False
self.training_interval = 60 # Train every 60 seconds
self.training_samples = []
self.max_training_samples = 1000
self.last_training_time = 0
self.last_predictions = {}
self.performance_metrics = {}
self.model_name = "enhanced_cnn_v1"
# Create checkpoint directory if it doesn't exist
os.makedirs(checkpoint_dir, exist_ok=True)
# Initialize CNN adapter
self._initialize_cnn_adapter()
logger.info(f"CNNDashboardIntegration initialized with checkpoint_dir: {checkpoint_dir}")
def _initialize_cnn_adapter(self):
"""Initialize the CNN adapter"""
try:
# Import here to avoid circular imports
from .enhanced_cnn_adapter import EnhancedCNNAdapter
# Create CNN adapter
self.cnn_adapter = EnhancedCNNAdapter(checkpoint_dir=self.checkpoint_dir)
# Load best checkpoint if available
self.cnn_adapter.load_best_checkpoint()
logger.info("CNN adapter initialized successfully")
except Exception as e:
logger.error(f"Error initializing CNN adapter: {e}")
self.cnn_adapter = None
def start_training_thread(self):
"""Start the training thread"""
if self.training_thread is not None and self.training_thread.is_alive():
logger.info("Training thread already running")
return
self.training_active = True
self.training_thread = threading.Thread(target=self._training_loop, daemon=True)
self.training_thread.start()
logger.info("CNN training thread started")
def stop_training_thread(self):
"""Stop the training thread"""
self.training_active = False
if self.training_thread is not None:
self.training_thread.join(timeout=5)
self.training_thread = None
logger.info("CNN training thread stopped")
def _training_loop(self):
"""Training loop for continuous model training"""
while self.training_active:
try:
# Check if it's time to train
current_time = time.time()
if current_time - self.last_training_time >= self.training_interval and len(self.training_samples) >= 10:
logger.info(f"Training CNN model with {len(self.training_samples)} samples")
# Train model
if self.cnn_adapter is not None:
metrics = self.cnn_adapter.train(epochs=1)
# Update performance metrics
self.performance_metrics = {
'loss': metrics.get('loss', 0.0),
'accuracy': metrics.get('accuracy', 0.0),
'samples': metrics.get('samples', 0),
'last_training': datetime.now().isoformat()
}
# Log training metrics
logger.info(f"CNN training metrics: loss={metrics.get('loss', 0.0):.4f}, accuracy={metrics.get('accuracy', 0.0):.4f}")
# Update last training time
self.last_training_time = current_time
# Sleep to avoid high CPU usage
time.sleep(1)
except Exception as e:
logger.error(f"Error in CNN training loop: {e}")
time.sleep(5) # Sleep longer on error
def process_data(self, symbol: str, base_data: BaseDataInput) -> Optional[ModelOutput]:
"""
Process data for model inference and training
Args:
symbol: Trading symbol
base_data: Standardized input data
Returns:
Optional[ModelOutput]: Model output, or None if processing failed
"""
try:
if self.cnn_adapter is None:
logger.warning("CNN adapter not initialized")
return None
# Make prediction
model_output = self.cnn_adapter.predict(base_data)
# Store prediction
self.last_predictions[symbol] = model_output
# Store model output in data provider
if self.data_provider is not None:
self.data_provider.store_model_output(model_output)
return model_output
except Exception as e:
logger.error(f"Error processing data for CNN model: {e}")
return None
def add_training_sample(self, base_data: BaseDataInput, actual_action: str, reward: float):
"""
Add a training sample
Args:
base_data: Standardized input data
actual_action: Actual action taken ('BUY', 'SELL', 'HOLD')
reward: Reward received for the action
"""
try:
if self.cnn_adapter is None:
logger.warning("CNN adapter not initialized")
return
# Add training sample to CNN adapter
self.cnn_adapter.add_training_sample(base_data, actual_action, reward)
# Add to local training samples
self.training_samples.append((base_data.symbol, actual_action, reward))
# Limit training samples
if len(self.training_samples) > self.max_training_samples:
self.training_samples = self.training_samples[-self.max_training_samples:]
logger.debug(f"Added training sample for {base_data.symbol}, action: {actual_action}, reward: {reward:.4f}")
except Exception as e:
logger.error(f"Error adding training sample: {e}")
def get_performance_metrics(self) -> Dict[str, Any]:
"""
Get performance metrics
Returns:
Dict[str, Any]: Performance metrics
"""
metrics = self.performance_metrics.copy()
# Add additional metrics
metrics['training_samples'] = len(self.training_samples)
metrics['model_name'] = self.model_name
# Add last prediction metrics
if self.last_predictions:
for symbol, prediction in self.last_predictions.items():
metrics[f'{symbol}_last_action'] = prediction.predictions.get('action', 'UNKNOWN')
metrics[f'{symbol}_last_confidence'] = prediction.confidence
return metrics
def get_visualization_data(self, symbol: str) -> Dict[str, Any]:
"""
Get visualization data for the dashboard
Args:
symbol: Trading symbol
Returns:
Dict[str, Any]: Visualization data
"""
data = {
'model_name': self.model_name,
'symbol': symbol,
'timestamp': datetime.now().isoformat(),
'performance_metrics': self.get_performance_metrics()
}
# Add last prediction
if symbol in self.last_predictions:
prediction = self.last_predictions[symbol]
data['last_prediction'] = {
'action': prediction.predictions.get('action', 'UNKNOWN'),
'confidence': prediction.confidence,
'timestamp': prediction.timestamp.isoformat(),
'buy_probability': prediction.predictions.get('buy_probability', 0.0),
'sell_probability': prediction.predictions.get('sell_probability', 0.0),
'hold_probability': prediction.predictions.get('hold_probability', 0.0)
}
# Add training samples summary
symbol_samples = [s for s in self.training_samples if s[0] == symbol]
data['training_samples'] = {
'total': len(symbol_samples),
'buy': len([s for s in symbol_samples if s[1] == 'BUY']),
'sell': len([s for s in symbol_samples if s[1] == 'SELL']),
'hold': len([s for s in symbol_samples if s[1] == 'HOLD']),
'avg_reward': sum(s[2] for s in symbol_samples) / len(symbol_samples) if symbol_samples else 0.0
}
return data
# Global CNN dashboard integration instance
_cnn_dashboard_integration = None
def get_cnn_dashboard_integration(data_provider=None) -> CNNDashboardIntegration:
"""
Get the global CNN dashboard integration instance
Args:
data_provider: Data provider instance
Returns:
CNNDashboardIntegration: Global CNN dashboard integration instance
"""
global _cnn_dashboard_integration
if _cnn_dashboard_integration is None:
_cnn_dashboard_integration = CNNDashboardIntegration(data_provider=data_provider)
return _cnn_dashboard_integration

13
core/cob_integration.py
View File

@ -101,9 +101,20 @@ class COBIntegration:
# Initialize COB provider as fallback # Initialize COB provider as fallback
try: try:
# Create default exchange configs
exchange_configs = {
'binance': {
'name': 'binance',
'enabled': True,
'websocket_url': 'wss://stream.binance.com:9443/ws/',
'rest_api_url': 'https://api.binance.com/api/v3/',
'rate_limits': {'requests_per_minute': 1200}
}
}
self.cob_provider = MultiExchangeCOBProvider( self.cob_provider = MultiExchangeCOBProvider(
symbols=self.symbols, symbols=self.symbols,
bucket_size_bps=1.0 # 1 basis point granularity exchange_configs=exchange_configs
) )
# Register callbacks # Register callbacks

1151
core/data_provider.py
View File

File diff suppressed because it is too large Load Diff

528
core/orchestrator.py
View File

@ -214,13 +214,8 @@ class TradingOrchestrator:
# Training tracking # Training tracking
self.last_trained_symbols: Dict[str, datetime] = {} self.last_trained_symbols: Dict[str, datetime] = {}
# INFERENCE DATA STORAGE - Per-model storage with memory optimization # SIMPLIFIED INFERENCE DATA STORAGE - Single last inference per model
self.inference_history: Dict[str, deque] = {} # {model_name: deque of last 5 inference records} self.last_inference: Dict[str, Dict] = {} # {model_name: last_inference_record}
self.max_memory_inferences = 5 # Keep only last 5 inferences in memory per model
self.max_disk_files_per_model = 200 # Cap disk files per model
# Initialize inference history for each model (will be populated as models make predictions)
# We'll create entries dynamically as models are used
# Initialize inference logger # Initialize inference logger
self.inference_logger = get_inference_logger() self.inference_logger = get_inference_logger()
@ -240,10 +235,16 @@ class TradingOrchestrator:
logger.info(f"Primary symbol: {self.symbol}, Reference symbols: {self.ref_symbols}") logger.info(f"Primary symbol: {self.symbol}, Reference symbols: {self.ref_symbols}")
logger.info("Universal Data Adapter integrated for centralized data flow") logger.info("Universal Data Adapter integrated for centralized data flow")
# Start centralized data collection for all models and dashboard # Start data collection if available
logger.info("Starting centralized data collection...") logger.info("Starting data collection...")
if hasattr(self.data_provider, 'start_centralized_data_collection'):
self.data_provider.start_centralized_data_collection() self.data_provider.start_centralized_data_collection()
logger.info("Centralized data collection started - all models and dashboard will receive data") logger.info("Centralized data collection started - all models and dashboard will receive data")
elif hasattr(self.data_provider, 'start_training_data_collection'):
self.data_provider.start_training_data_collection()
logger.info("Training data collection started")
else:
logger.info("Data provider does not require explicit data collection startup")
# Data provider is already initialized and optimized # Data provider is already initialized and optimized
@ -683,13 +684,10 @@ class TradingOrchestrator:
self.sensitivity_learning_queue = [] self.sensitivity_learning_queue = []
self.perfect_move_buffer = [] self.perfect_move_buffer = []
# Clear inference history (but keep recent for training) # Clear any outcome evaluation flags for last inferences
for model_name in list(self.inference_history.keys()): for model_name in self.last_inference:
# Keep only the last inference for each model to maintain training capability if self.last_inference[model_name]:
if len(self.inference_history[model_name]) > 1: self.last_inference[model_name]['outcome_evaluated'] = False
last_inference = self.inference_history[model_name][-1]
self.inference_history[model_name].clear()
self.inference_history[model_name].append(last_inference)
# Clear fusion training data # Clear fusion training data
self.fusion_training_data = [] self.fusion_training_data = []
@ -1114,10 +1112,10 @@ class TradingOrchestrator:
if model.name not in self.model_performance: if model.name not in self.model_performance:
self.model_performance[model.name] = {'correct': 0, 'total': 0, 'accuracy': 0.0} self.model_performance[model.name] = {'correct': 0, 'total': 0, 'accuracy': 0.0}
# Initialize inference history for this model # Initialize last inference storage for this model
if model.name not in self.inference_history: if model.name not in self.last_inference:
self.inference_history[model.name] = deque(maxlen=self.max_memory_inferences) self.last_inference[model.name] = None
logger.debug(f"Initialized inference history for {model.name}") logger.debug(f"Initialized last inference storage for {model.name}")
logger.info(f"Registered {model.name} model with weight {self.model_weights[model.name]}") logger.info(f"Registered {model.name} model with weight {self.model_weights[model.name]}")
self._normalize_weights() self._normalize_weights()
@ -1320,12 +1318,7 @@ class TradingOrchestrator:
logger.error(f"Error getting prediction from {model_name}: {e}") logger.error(f"Error getting prediction from {model_name}: {e}")
continue continue
# Debug: Log inference history status (only if low record count)
total_records = sum(len(history) for history in self.inference_history.values())
if total_records < 10: # Only log when we have few records
logger.debug(f"Total inference records across all models: {total_records}")
for model_name, history in self.inference_history.items():
logger.debug(f" {model_name}: {len(history)} records")
# Trigger training based on previous inference data # Trigger training based on previous inference data
await self._trigger_model_training(symbol) await self._trigger_model_training(symbol)
@ -1392,17 +1385,15 @@ class TradingOrchestrator:
return {} return {}
async def _store_inference_data_async(self, model_name: str, model_input: Any, prediction: Prediction, timestamp: datetime, symbol: str = None): async def _store_inference_data_async(self, model_name: str, model_input: Any, prediction: Prediction, timestamp: datetime, symbol: str = None):
"""Store inference data per-model with async file operations and memory optimization""" """Store last inference in memory and all inferences to database for future training"""
try: try:
# Only log first few inference records to avoid spam logger.debug(f"Storing inference for {model_name}: {prediction.action} (confidence: {prediction.confidence:.3f})")
if len(self.inference_history.get(model_name, [])) < 3:
logger.debug(f"Storing inference data for {model_name}: {prediction.action} (confidence: {prediction.confidence:.3f})")
# Extract symbol from prediction if not provided # Extract symbol from prediction if not provided
if symbol is None: if symbol is None:
symbol = getattr(prediction, 'symbol', 'ETH/USDT') # Default to ETH/USDT if not available symbol = getattr(prediction, 'symbol', 'ETH/USDT') # Default to ETH/USDT if not available
# Create comprehensive inference record # Create inference record - store only what's needed for training
inference_record = { inference_record = {
'timestamp': timestamp.isoformat(), 'timestamp': timestamp.isoformat(),
'symbol': symbol, 'symbol': symbol,
@ -1414,229 +1405,155 @@ class TradingOrchestrator:
'probabilities': prediction.probabilities, 'probabilities': prediction.probabilities,
'timeframe': prediction.timeframe 'timeframe': prediction.timeframe
}, },
'metadata': prediction.metadata or {} 'metadata': prediction.metadata or {},
'training_outcome': None, # Will be set when training occurs
'outcome_evaluated': False
} }
# Store in memory (only last 5 per model) # Store only the last inference per model (for immediate training)
if model_name not in self.inference_history: self.last_inference[model_name] = inference_record
self.inference_history[model_name] = deque(maxlen=self.max_memory_inferences)
self.inference_history[model_name].append(inference_record) # Also save to database using database manager for future training and analysis
asyncio.create_task(self._save_to_database_manager_async(model_name, inference_record))
# Async file storage (don't wait for completion) logger.debug(f"Stored last inference for {model_name} and queued database save")
asyncio.create_task(self._save_inference_to_disk_async(model_name, inference_record))
logger.debug(f"Stored inference data for {model_name} (memory: {len(self.inference_history[model_name])}/5)")
except Exception as e: except Exception as e:
logger.error(f"Error storing inference data for {model_name}: {e}") logger.error(f"Error storing inference data for {model_name}: {e}")
async def _save_inference_to_disk_async(self, model_name: str, inference_record: Dict): async def _save_to_database_manager_async(self, model_name: str, inference_record: Dict):
"""Async save inference record to SQLite database and model-specific log""" """Save inference record using DatabaseManager for future training"""
try: import hashlib
# Use SQLite for comprehensive storage
await self._save_to_sqlite_db(model_name, inference_record)
# Also save key metrics to model-specific log for debugging
await self._save_to_model_log(model_name, inference_record)
except Exception as e:
logger.error(f"Error saving inference to disk for {model_name}: {e}")
async def _save_to_sqlite_db(self, model_name: str, inference_record: Dict):
"""Save inference record to SQLite database"""
import sqlite3
import asyncio import asyncio
def save_to_db(): def save_to_db():
try: try:
# Create database directory
db_dir = Path("training_data/inference_db")
db_dir.mkdir(parents=True, exist_ok=True)
# Connect to SQLite database
db_path = db_dir / "inference_history.db"
conn = sqlite3.connect(str(db_path))
cursor = conn.cursor()
# Create table if it doesn't exist
cursor.execute('''
CREATE TABLE IF NOT EXISTS inference_records (
id INTEGER PRIMARY KEY AUTOINCREMENT,
model_name TEXT NOT NULL,
symbol TEXT NOT NULL,
timestamp TEXT NOT NULL,
action TEXT NOT NULL,
confidence REAL NOT NULL,
probabilities TEXT,
timeframe TEXT,
metadata TEXT,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
)
''')
# Create index for faster queries
cursor.execute('''
CREATE INDEX IF NOT EXISTS idx_model_timestamp
ON inference_records(model_name, timestamp)
''')
# Extract data from inference record # Extract data from inference record
prediction = inference_record.get('prediction', {}) prediction = inference_record.get('prediction', {})
probabilities_str = str(prediction.get('probabilities', {})) symbol = inference_record.get('symbol', 'ETH/USDT')
metadata_str = str(inference_record.get('metadata', {})) timestamp_str = inference_record.get('timestamp', '')
# Insert record # Parse timestamp
cursor.execute(''' if isinstance(timestamp_str, str):
INSERT INTO inference_records timestamp = datetime.fromisoformat(timestamp_str)
(model_name, symbol, timestamp, action, confidence, probabilities, timeframe, metadata) else:
VALUES (?, ?, ?, ?, ?, ?, ?, ?) timestamp = timestamp_str
''', (
model_name,
inference_record.get('symbol', 'ETH/USDT'),
inference_record.get('timestamp', ''),
prediction.get('action', 'HOLD'),
prediction.get('confidence', 0.0),
probabilities_str,
prediction.get('timeframe', '1m'),
metadata_str
))
# Clean up old records (keep only last 1000 per model) # Create hash of input features for deduplication
cursor.execute(''' model_input = inference_record.get('model_input')
DELETE FROM inference_records input_features_hash = "unknown"
WHERE model_name = ? AND id NOT IN ( input_features_array = None
SELECT id FROM inference_records
WHERE model_name = ? if model_input is not None:
ORDER BY timestamp DESC # Convert to numpy array if possible
LIMIT 1000 try:
if hasattr(model_input, 'numpy'): # PyTorch tensor
input_features_array = model_input.detach().cpu().numpy()
elif isinstance(model_input, np.ndarray):
input_features_array = model_input
elif isinstance(model_input, (list, tuple)):
input_features_array = np.array(model_input)
# Create hash of the input features
if input_features_array is not None:
input_features_hash = hashlib.md5(input_features_array.tobytes()).hexdigest()[:16]
except Exception as e:
logger.debug(f"Could not process input features for hashing: {e}")
# Create InferenceRecord using the database manager's structure
from utils.database_manager import InferenceRecord
db_record = InferenceRecord(
model_name=model_name,
timestamp=timestamp,
symbol=symbol,
action=prediction.get('action', 'HOLD'),
confidence=prediction.get('confidence', 0.0),
probabilities=prediction.get('probabilities', {}),
input_features_hash=input_features_hash,
processing_time_ms=0.0, # We don't track this in orchestrator
memory_usage_mb=0.0, # We don't track this in orchestrator
input_features=input_features_array,
checkpoint_id=None,
metadata=inference_record.get('metadata', {})
) )
''', (model_name, model_name))
conn.commit() # Log using database manager
conn.close() success = self.db_manager.log_inference(db_record)
if success:
logger.debug(f"Saved inference to database for {model_name}")
else:
logger.warning(f"Failed to save inference to database for {model_name}")
except Exception as e: except Exception as e:
logger.error(f"Error saving to SQLite database: {e}") logger.error(f"Error saving to database manager: {e}")
# Run database operation in thread pool to avoid blocking # Run database operation in thread pool to avoid blocking
await asyncio.get_event_loop().run_in_executor(None, save_to_db) await asyncio.get_event_loop().run_in_executor(None, save_to_db)
async def _save_to_model_log(self, model_name: str, inference_record: Dict):
"""Save key inference metrics to model-specific log file for debugging"""
import asyncio
def save_to_log():
def get_last_inference_status(self) -> Dict[str, Any]:
"""Get status of last inferences for all models"""
status = {}
for model_name, inference in self.last_inference.items():
if inference:
status[model_name] = {
'timestamp': inference.get('timestamp'),
'symbol': inference.get('symbol'),
'action': inference.get('prediction', {}).get('action'),
'confidence': inference.get('prediction', {}).get('confidence'),
'outcome_evaluated': inference.get('outcome_evaluated', False),
'training_outcome': inference.get('training_outcome')
}
else:
status[model_name] = None
return status
def get_training_data_from_db(self, model_name: str, symbol: str = None, hours_back: int = 24, limit: int = 1000) -> List[Dict]:
"""Get inference records for training from database manager"""
try: try:
# Create logs directory # Use database manager's method specifically for training data
logs_dir = Path("logs/model_inference") db_records = self.db_manager.get_inference_records_for_training(
logs_dir.mkdir(parents=True, exist_ok=True) model_name=model_name,
symbol=symbol,
# Create model-specific log file hours_back=hours_back,
log_file = logs_dir / f"{model_name}_inference.log" limit=limit
# Extract key metrics
prediction = inference_record.get('prediction', {})
timestamp = inference_record.get('timestamp', '')
symbol = inference_record.get('symbol', 'N/A')
# Format log entry with key metrics
log_entry = (
f"{timestamp} | "
f"Symbol: {symbol} | "
f"Action: {prediction.get('action', 'N/A'):4} | "
f"Confidence: {prediction.get('confidence', 0.0):6.3f} | "
f"Timeframe: {prediction.get('timeframe', 'N/A'):3} | "
f"Probs: BUY={prediction.get('probabilities', {}).get('BUY', 0.0):5.3f} "
f"SELL={prediction.get('probabilities', {}).get('SELL', 0.0):5.3f} "
f"HOLD={prediction.get('probabilities', {}).get('HOLD', 0.0):5.3f}\n"
) )
# Append to log file # Convert to our format
with open(log_file, 'a', encoding='utf-8') as f:
f.write(log_entry)
# Keep log files manageable (rotate when > 10MB)
if log_file.stat().st_size > 10 * 1024 * 1024: # 10MB
self._rotate_log_file(log_file)
except Exception as e:
logger.error(f"Error saving to model log: {e}")
# Run log operation in thread pool to avoid blocking
await asyncio.get_event_loop().run_in_executor(None, save_to_log)
def _rotate_log_file(self, log_file: Path):
"""Rotate log file when it gets too large"""
try:
# Keep last 1000 lines
with open(log_file, 'r', encoding='utf-8') as f:
lines = f.readlines()
# Write back only the last 1000 lines
with open(log_file, 'w', encoding='utf-8') as f:
f.writelines(lines[-1000:])
logger.debug(f"Rotated log file {log_file.name} (kept last 1000 lines)")
except Exception as e:
logger.error(f"Error rotating log file {log_file}: {e}")
def get_inference_records_from_db(self, model_name: str = None, limit: int = 100) -> List[Dict]:
"""Get inference records from SQLite database"""
import sqlite3
try:
# Connect to database
db_path = Path("training_data/inference_db/inference_history.db")
if not db_path.exists():
return []
conn = sqlite3.connect(str(db_path))
cursor = conn.cursor()
# Query records
if model_name:
cursor.execute('''
SELECT model_name, symbol, timestamp, action, confidence, probabilities, timeframe, metadata
FROM inference_records
WHERE model_name = ?
ORDER BY timestamp DESC
LIMIT ?
''', (model_name, limit))
else:
cursor.execute('''
SELECT model_name, symbol, timestamp, action, confidence, probabilities, timeframe, metadata
FROM inference_records
ORDER BY timestamp DESC
LIMIT ?
''', (limit,))
records = [] records = []
for row in cursor.fetchall(): for db_record in db_records:
try:
record = { record = {
'model_name': row[0], 'model_name': db_record.model_name,
'symbol': row[1], 'symbol': db_record.symbol,
'timestamp': row[2], 'timestamp': db_record.timestamp.isoformat(),
'prediction': { 'prediction': {
'action': row[3], 'action': db_record.action,
'confidence': row[4], 'confidence': db_record.confidence,
'probabilities': eval(row[5]) if row[5] else {}, 'probabilities': db_record.probabilities,
'timeframe': row[6] 'timeframe': '1m'
}, },
'metadata': eval(row[7]) if row[7] else {} 'metadata': db_record.metadata or {},
'model_input': db_record.input_features, # Full input features for training
'input_features_hash': db_record.input_features_hash
} }
records.append(record) records.append(record)
except Exception as e:
logger.warning(f"Skipping malformed training record: {e}")
continue
conn.close() logger.info(f"Retrieved {len(records)} training records for {model_name}")
return records return records
except Exception as e: except Exception as e:
logger.error(f"Error querying SQLite database: {e}") logger.error(f"Error getting training data from database: {e}")
return [] return []
def _prepare_cnn_input_data(self, ohlcv_data: Dict, cob_data: Any, technical_indicators: Dict) -> torch.Tensor: def _prepare_cnn_input_data(self, ohlcv_data: Dict, cob_data: Any, technical_indicators: Dict) -> torch.Tensor:
"""Prepare standardized input data for CNN models with proper GPU device placement""" """Prepare standardized input data for CNN models with proper GPU device placement"""
try: try:
@ -1763,197 +1680,58 @@ class TradingOrchestrator:
'outcome_evaluated': False 'outcome_evaluated': False
} }
# Store in memory (inference history) - keyed by model_name # Store only the last inference per model (for immediate training)
if model_name not in self.inference_history: self.last_inference[model_name] = inference_record
self.inference_history[model_name] = deque(maxlen=self.max_memory_inferences)
self.inference_history[model_name].append(inference_record) # Also save to database using database manager for future training (run in background)
logger.debug(f"Stored inference data for {model_name} on {symbol}") asyncio.create_task(self._save_to_database_manager_async(model_name, inference_record))
# Persistent storage to disk (for long-term training data) logger.debug(f"Stored last inference for {model_name} on {symbol} and queued database save")
self._save_inference_to_disk(inference_record)
except Exception as e: except Exception as e:
logger.error(f"Error storing inference data: {e}") logger.error(f"Error storing inference data: {e}")
def _save_inference_to_disk(self, inference_record: Dict):
"""Save inference record to persistent storage"""
try:
# Create inference data directory
inference_dir = Path("training_data/inference_history")
inference_dir.mkdir(parents=True, exist_ok=True)
# Create filename with timestamp and model name
timestamp_str = inference_record['timestamp'].strftime('%Y%m%d_%H%M%S')
filename = f"{inference_record['symbol']}_{inference_record['model_name']}_{timestamp_str}.json"
filepath = inference_dir / filename
# Convert numpy arrays to lists for JSON serialization
serializable_record = self._make_json_serializable(inference_record)
# Save to JSON file
with open(filepath, 'w') as f:
json.dump(serializable_record, f, indent=2)
logger.debug(f"Saved inference record to disk: {filepath}")
except Exception as e:
logger.error(f"Error saving inference to disk: {e}")
def _make_json_serializable(self, obj):
"""Convert object to JSON-serializable format"""
if isinstance(obj, dict):
return {k: self._make_json_serializable(v) for k, v in obj.items()}
elif isinstance(obj, list):
return [self._make_json_serializable(item) for item in obj]
elif isinstance(obj, np.ndarray):
return obj.tolist()
elif isinstance(obj, (np.integer, np.floating)):
return obj.item()
elif isinstance(obj, datetime):
return obj.isoformat()
else:
return obj
def load_inference_history_from_disk(self, symbol: str, days_back: int = 7) -> List[Dict]:
"""Load inference history from SQLite database for training replay"""
try:
import sqlite3
# Connect to database
db_path = Path("training_data/inference_db/inference_history.db")
if not db_path.exists():
return []
conn = sqlite3.connect(str(db_path))
cursor = conn.cursor()
# Get records for the symbol from the last N days
cutoff_date = (datetime.now() - timedelta(days=days_back)).isoformat()
cursor.execute('''
SELECT model_name, symbol, timestamp, action, confidence, probabilities, timeframe, metadata
FROM inference_records
WHERE symbol = ? AND timestamp >= ?
ORDER BY timestamp ASC
''', (symbol, cutoff_date))
inference_records = []
for row in cursor.fetchall():
record = {
'model_name': row[0],
'symbol': row[1],
'timestamp': row[2],
'prediction': {
'action': row[3],
'confidence': row[4],
'probabilities': eval(row[5]) if row[5] else {},
'timeframe': row[6]
},
'metadata': eval(row[7]) if row[7] else {}
}
inference_records.append(record)
conn.close()
logger.info(f"Loaded {len(inference_records)} inference records for {symbol} from SQLite database")
return inference_records
except Exception as e:
logger.error(f"Error loading inference history from database: {e}")
return []
async def load_model_inference_history(self, model_name: str, limit: int = 50) -> List[Dict]:
"""Load inference history for a specific model from SQLite database"""
try:
# Use the SQLite database method
records = self.get_inference_records_from_db(model_name, limit)
logger.info(f"Loaded {len(records)} inference records for {model_name} from database")
return records
except Exception as e:
logger.error(f"Error loading model inference history for {model_name}: {e}")
return []
def get_model_training_data(self, model_name: str, symbol: str = None) -> List[Dict]: def get_model_training_data(self, model_name: str, symbol: str = None) -> List[Dict]:
"""Get training data for a specific model""" """Get training data for a specific model"""
try: try:
training_data = [] training_data = []
# Get from memory first # Use database manager to get training data
if symbol: training_data = self.get_training_data_from_db(model_name, symbol)
symbols_to_check = [symbol]
else:
symbols_to_check = self.symbols
for sym in symbols_to_check: logger.info(f"Retrieved {len(training_data)} training records for {model_name}")
if sym in self.inference_history: return training_data
for record in self.inference_history[sym]:
if record['model_name'] == model_name:
training_data.append(record)
# Also load from disk for more comprehensive training data
for sym in symbols_to_check:
disk_records = self.load_inference_history_from_disk(sym)
for record in disk_records:
if record['model_name'] == model_name:
training_data.append(record)
# Remove duplicates and sort by timestamp
seen_timestamps = set()
unique_data = []
for record in training_data:
timestamp_key = f"{record['timestamp']}_{record['symbol']}"
if timestamp_key not in seen_timestamps:
seen_timestamps.add(timestamp_key)
unique_data.append(record)
unique_data.sort(key=lambda x: x['timestamp'])
logger.info(f"Retrieved {len(unique_data)} training records for {model_name}")
return unique_data
except Exception as e: except Exception as e:
logger.error(f"Error getting model training data: {e}") logger.error(f"Error getting model training data: {e}")
return [] return []
async def _trigger_model_training(self, symbol: str): async def _trigger_model_training(self, symbol: str):
"""Trigger training for models based on previous inference data""" """Trigger training for models based on their last inference"""
try: try:
if not self.training_enabled: if not self.training_enabled:
logger.debug("Training disabled, skipping model training") logger.debug("Training disabled, skipping model training")
return return
# Check if we have any inference history for any model # Check if we have any last inferences for any model
if not self.inference_history: if not self.last_inference:
logger.debug("No inference history available for training") logger.debug("No inference data available for training")
return return
# Get recent inference records from all models (not symbol-based)
all_recent_records = []
for model_name, model_records in self.inference_history.items():
all_recent_records.extend(list(model_records))
# Only log if we have few records (for debugging)
if len(all_recent_records) < 5:
logger.debug(f"Total inference records for training: {len(all_recent_records)}")
for model_name, model_records in self.inference_history.items():
logger.debug(f" Model {model_name} has {len(model_records)} inference records")
if len(all_recent_records) < 2:
logger.debug("Not enough inference records for training")
return # Need at least 2 records to compare
# Get current price for outcome evaluation # Get current price for outcome evaluation
current_price = self.data_provider.get_current_price(symbol) current_price = self.data_provider.get_current_price(symbol)
if current_price is None: if current_price is None:
return return
# Train on the most recent inference record (last prediction made) # Train each model based on its last inference
if all_recent_records: for model_name, last_inference_record in self.last_inference.items():
# Get the most recent record for training if last_inference_record and not last_inference_record.get('outcome_evaluated', False):
most_recent_record = max(all_recent_records, key=lambda x: datetime.fromisoformat(x['timestamp']) if isinstance(x['timestamp'], str) else x['timestamp']) await self._evaluate_and_train_on_record(last_inference_record, current_price)
await self._evaluate_and_train_on_record(most_recent_record, current_price)
except Exception as e: except Exception as e:
logger.error(f"Error triggering model training for {symbol}: {e}") logger.error(f"Error triggering model training for {symbol}: {e}")
@ -2011,6 +1789,16 @@ class TradingOrchestrator:
# Train the specific model based on sophisticated outcome # Train the specific model based on sophisticated outcome
await self._train_model_on_outcome(record, was_correct, price_change_pct, reward) await self._train_model_on_outcome(record, was_correct, price_change_pct, reward)
# Mark this inference as evaluated to prevent re-training
if model_name in self.last_inference and self.last_inference[model_name] == record:
self.last_inference[model_name]['outcome_evaluated'] = True
self.last_inference[model_name]['training_outcome'] = {
'was_correct': was_correct,
'reward': reward,
'price_change_pct': price_change_pct,
'evaluated_at': datetime.now().isoformat()
}
logger.debug(f"Evaluated {model_name} prediction: {'✓' if was_correct else '✗'} " logger.debug(f"Evaluated {model_name} prediction: {'✓' if was_correct else '✗'} "
f"({prediction['action']}, {price_change_pct:.2f}% change, " f"({prediction['action']}, {price_change_pct:.2f}% change, "
f"confidence: {prediction_confidence:.3f}, reward: {reward:.3f})") f"confidence: {prediction_confidence:.3f}, reward: {reward:.3f})")
@ -2215,7 +2003,7 @@ class TradingOrchestrator:
) )
predictions.append(prediction) predictions.append(prediction)
# Store prediction in SQLite database for training # Store prediction in database for training
logger.debug(f"Added CNN prediction to database: {prediction}") logger.debug(f"Added CNN prediction to database: {prediction}")
# Note: Inference data will be stored in main prediction loop to avoid duplication # Note: Inference data will be stored in main prediction loop to avoid duplication

BIN
data/trading_system.db
View File

Binary file not shown.

4980
dataprovider_realtime.py
View File

File diff suppressed because it is too large Load Diff

89
example_usage_simplified_data_provider.py Normal file
View File

@ -0,0 +1,89 @@
#!/usr/bin/env python3
"""
Example usage of the simplified data provider
"""
import time
import logging
from core.data_provider import DataProvider
# Set up logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
def main():
"""Demonstrate the simplified data provider usage"""
# Initialize data provider (starts automatic maintenance)
logger.info("Initializing DataProvider...")
dp = DataProvider()
# Wait for initial data load (happens automatically in background)
logger.info("Waiting for initial data load...")
time.sleep(15) # Give it time to load data
# Example 1: Get cached historical data (no API calls)
logger.info("\n=== Example 1: Getting Historical Data ===")
eth_1m_data = dp.get_historical_data('ETH/USDT', '1m', limit=50)
if eth_1m_data is not None:
logger.info(f"ETH/USDT 1m data: {len(eth_1m_data)} candles")
logger.info(f"Latest candle: {eth_1m_data.iloc[-1]['close']}")
# Example 2: Get current prices
logger.info("\n=== Example 2: Current Prices ===")
eth_price = dp.get_current_price('ETH/USDT')
btc_price = dp.get_current_price('BTC/USDT')
logger.info(f"ETH current price: ${eth_price}")
logger.info(f"BTC current price: ${btc_price}")
# Example 3: Check cache status
logger.info("\n=== Example 3: Cache Status ===")
cache_summary = dp.get_cached_data_summary()
for symbol in cache_summary['cached_data']:
logger.info(f"\n{symbol}:")
for timeframe, info in cache_summary['cached_data'][symbol].items():
if 'candle_count' in info and info['candle_count'] > 0:
logger.info(f" {timeframe}: {info['candle_count']} candles, latest: ${info['latest_price']}")
else:
logger.info(f" {timeframe}: {info.get('status', 'no data')}")
# Example 4: Multiple timeframe data
logger.info("\n=== Example 4: Multiple Timeframes ===")
for tf in ['1s', '1m', '1h', '1d']:
data = dp.get_historical_data('ETH/USDT', tf, limit=5)
if data is not None and not data.empty:
logger.info(f"ETH {tf}: {len(data)} candles, range: ${data['close'].min():.2f} - ${data['close'].max():.2f}")
# Example 5: Health check
logger.info("\n=== Example 5: Health Check ===")
health = dp.health_check()
logger.info(f"Data maintenance active: {health['data_maintenance_active']}")
logger.info(f"Symbols: {health['symbols']}")
logger.info(f"Timeframes: {health['timeframes']}")
# Example 6: Wait and show automatic updates
logger.info("\n=== Example 6: Automatic Updates ===")
logger.info("Waiting 30 seconds to show automatic data updates...")
# Get initial timestamp
initial_data = dp.get_historical_data('ETH/USDT', '1s', limit=1)
initial_time = initial_data.index[-1] if initial_data is not None else None
time.sleep(30)
# Check if data was updated
updated_data = dp.get_historical_data('ETH/USDT', '1s', limit=1)
updated_time = updated_data.index[-1] if updated_data is not None else None
if initial_time and updated_time and updated_time > initial_time:
logger.info(f"✅ Data automatically updated! New timestamp: {updated_time}")
else:
logger.info("⏳ Data update in progress...")
# Clean shutdown
logger.info("\n=== Shutting Down ===")
dp.stop_automatic_data_maintenance()
logger.info("DataProvider stopped successfully")
if __name__ == "__main__":
main()

118
test_cob_data_quality.py Normal file
View File

@ -0,0 +1,118 @@
#!/usr/bin/env python3
"""
Test script for COB data quality and imbalance indicators
"""
import time
import logging
from core.data_provider import DataProvider
# Set up logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
def test_cob_data_quality():
"""Test COB data quality and imbalance indicators"""
logger.info("Testing COB data quality and imbalance indicators...")
# Initialize data provider
dp = DataProvider()
# Wait for initial data load and COB connection
logger.info("Waiting for initial data load and COB connection...")
time.sleep(20)
# Test 1: Check cached data summary
logger.info("\n=== Test 1: Cached Data Summary ===")
cache_summary = dp.get_cached_data_summary()
for symbol in cache_summary['cached_data']:
logger.info(f"\n{symbol}:")
for timeframe, info in cache_summary['cached_data'][symbol].items():
if 'candle_count' in info and info['candle_count'] > 0:
logger.info(f" {timeframe}: {info['candle_count']} candles, latest: ${info['latest_price']}")
else:
logger.info(f" {timeframe}: {info.get('status', 'no data')}")
# Test 2: Check COB data quality
logger.info("\n=== Test 2: COB Data Quality ===")
cob_quality = dp.get_cob_data_quality()
for symbol in cob_quality['symbols']:
logger.info(f"\n{symbol} COB Data:")
# Raw ticks
raw_info = cob_quality['raw_ticks'].get(symbol, {})
logger.info(f" Raw ticks: {raw_info.get('count', 0)} ticks")
if raw_info.get('age_seconds') is not None:
logger.info(f" Raw data age: {raw_info['age_seconds']:.1f} seconds")
# Aggregated 1s data
agg_info = cob_quality['aggregated_1s'].get(symbol, {})
logger.info(f" Aggregated 1s: {agg_info.get('count', 0)} records")
if agg_info.get('age_seconds') is not None:
logger.info(f" Aggregated data age: {agg_info['age_seconds']:.1f} seconds")
# Imbalance indicators
imbalance_info = cob_quality['imbalance_indicators'].get(symbol, {})
if imbalance_info:
logger.info(f" Imbalance 1s: {imbalance_info.get('imbalance_1s', 0):.4f}")
logger.info(f" Imbalance 5s: {imbalance_info.get('imbalance_5s', 0):.4f}")
logger.info(f" Imbalance 15s: {imbalance_info.get('imbalance_15s', 0):.4f}")
logger.info(f" Imbalance 60s: {imbalance_info.get('imbalance_60s', 0):.4f}")
logger.info(f" Total volume: {imbalance_info.get('total_volume', 0):.2f}")
logger.info(f" Price buckets: {imbalance_info.get('bucket_count', 0)}")
# Data freshness
freshness = cob_quality['data_freshness'].get(symbol, 'unknown')
logger.info(f" Data freshness: {freshness}")
# Test 3: Get recent COB aggregated data
logger.info("\n=== Test 3: Recent COB Aggregated Data ===")
for symbol in ['ETH/USDT', 'BTC/USDT']:
recent_cob = dp.get_cob_1s_aggregated(symbol, count=5)
logger.info(f"\n{symbol} - Last 5 aggregated records:")
for i, record in enumerate(recent_cob[-5:]):
timestamp = record.get('timestamp', 0)
imbalance_1s = record.get('imbalance_1s', 0)
imbalance_5s = record.get('imbalance_5s', 0)
total_volume = record.get('total_volume', 0)
bucket_count = len(record.get('bid_buckets', {})) + len(record.get('ask_buckets', {}))
logger.info(f" [{i+1}] Time: {timestamp}, Imb1s: {imbalance_1s:.4f}, "
f"Imb5s: {imbalance_5s:.4f}, Vol: {total_volume:.2f}, Buckets: {bucket_count}")
# Test 4: Monitor real-time updates
logger.info("\n=== Test 4: Real-time Updates (30 seconds) ===")
logger.info("Monitoring COB data updates...")
initial_quality = dp.get_cob_data_quality()
time.sleep(30)
updated_quality = dp.get_cob_data_quality()
for symbol in ['ETH/USDT', 'BTC/USDT']:
initial_count = initial_quality['raw_ticks'].get(symbol, {}).get('count', 0)
updated_count = updated_quality['raw_ticks'].get(symbol, {}).get('count', 0)
new_ticks = updated_count - initial_count
initial_agg = initial_quality['aggregated_1s'].get(symbol, {}).get('count', 0)
updated_agg = updated_quality['aggregated_1s'].get(symbol, {}).get('count', 0)
new_agg = updated_agg - initial_agg
logger.info(f"{symbol}: +{new_ticks} raw ticks, +{new_agg} aggregated records")
# Show latest imbalances
latest_imbalances = updated_quality['imbalance_indicators'].get(symbol, {})
if latest_imbalances:
logger.info(f" Latest imbalances: 1s={latest_imbalances.get('imbalance_1s', 0):.4f}, "
f"5s={latest_imbalances.get('imbalance_5s', 0):.4f}, "
f"15s={latest_imbalances.get('imbalance_15s', 0):.4f}, "
f"60s={latest_imbalances.get('imbalance_60s', 0):.4f}")
# Clean shutdown
logger.info("\n=== Shutting Down ===")
dp.stop_automatic_data_maintenance()
logger.info("COB data quality test completed")
if __name__ == "__main__":
test_cob_data_quality()

112
test_data_provider_integration.py Normal file
View File

@ -0,0 +1,112 @@
#!/usr/bin/env python3
"""
Integration test for the simplified data provider with other components
"""
import time
import logging
import pandas as pd
from core.data_provider import DataProvider
# Set up logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
def test_integration():
"""Test integration with other components"""
logger.info("Testing DataProvider integration...")
# Initialize data provider
dp = DataProvider()
# Wait for initial data load
logger.info("Waiting for initial data load...")
time.sleep(15)
# Test 1: Feature matrix generation
logger.info("\n=== Test 1: Feature Matrix Generation ===")
try:
feature_matrix = dp.get_feature_matrix('ETH/USDT', ['1m', '1h'], window_size=20)
if feature_matrix is not None:
logger.info(f"✅ Feature matrix shape: {feature_matrix.shape}")
else:
logger.warning("❌ Feature matrix generation failed")
except Exception as e:
logger.error(f"❌ Feature matrix error: {e}")
# Test 2: Multi-symbol data access
logger.info("\n=== Test 2: Multi-Symbol Data Access ===")
for symbol in ['ETH/USDT', 'BTC/USDT']:
for timeframe in ['1s', '1m', '1h', '1d']:
data = dp.get_historical_data(symbol, timeframe, limit=10)
if data is not None and not data.empty:
logger.info(f"✅ {symbol} {timeframe}: {len(data)} candles")
else:
logger.warning(f"❌ {symbol} {timeframe}: No data")
# Test 3: Data consistency checks
logger.info("\n=== Test 3: Data Consistency ===")
eth_1m = dp.get_historical_data('ETH/USDT', '1m', limit=100)
if eth_1m is not None and not eth_1m.empty:
# Check for proper OHLCV structure
required_cols = ['open', 'high', 'low', 'close', 'volume']
has_all_cols = all(col in eth_1m.columns for col in required_cols)
logger.info(f"✅ OHLCV columns present: {has_all_cols}")
# Check data types
numeric_cols = eth_1m[required_cols].dtypes
all_numeric = all(pd.api.types.is_numeric_dtype(dtype) for dtype in numeric_cols)
logger.info(f"✅ All columns numeric: {all_numeric}")
# Check for NaN values
has_nan = eth_1m[required_cols].isna().any().any()
logger.info(f"✅ No NaN values: {not has_nan}")
# Check price relationships (high >= low, etc.)
price_valid = (eth_1m['high'] >= eth_1m['low']).all()
logger.info(f"✅ Price relationships valid: {price_valid}")
# Test 4: Performance test
logger.info("\n=== Test 4: Performance Test ===")
start_time = time.time()
for i in range(100):
data = dp.get_historical_data('ETH/USDT', '1m', limit=50)
end_time = time.time()
avg_time = (end_time - start_time) / 100 * 1000 # ms
logger.info(f"✅ Average data access time: {avg_time:.2f}ms")
# Test 5: Current price accuracy
logger.info("\n=== Test 5: Current Price Accuracy ===")
eth_price = dp.get_current_price('ETH/USDT')
eth_data = dp.get_historical_data('ETH/USDT', '1s', limit=1)
if eth_data is not None and not eth_data.empty:
latest_close = eth_data.iloc[-1]['close']
price_match = abs(eth_price - latest_close) < 0.01
logger.info(f"✅ Current price matches latest candle: {price_match}")
logger.info(f" Current price: ${eth_price}")
logger.info(f" Latest close: ${latest_close}")
# Test 6: Cache efficiency
logger.info("\n=== Test 6: Cache Efficiency ===")
cache_summary = dp.get_cached_data_summary()
total_candles = 0
for symbol_data in cache_summary['cached_data'].values():
for tf_data in symbol_data.values():
if isinstance(tf_data, dict) and 'candle_count' in tf_data:
total_candles += tf_data['candle_count']
logger.info(f"✅ Total cached candles: {total_candles}")
logger.info(f"✅ Data maintenance active: {cache_summary['data_maintenance_active']}")
# Test 7: Memory usage estimation
logger.info("\n=== Test 7: Memory Usage Estimation ===")
# Rough estimation: 8 columns * 8 bytes * total_candles
estimated_memory_mb = (total_candles * 8 * 8) / (1024 * 1024)
logger.info(f"✅ Estimated memory usage: {estimated_memory_mb:.2f} MB")
# Clean shutdown
dp.stop_automatic_data_maintenance()
logger.info("\n✅ Integration test completed successfully!")
if __name__ == "__main__":
test_integration()

118
test_imbalance_calculation.py Normal file
View File

@ -0,0 +1,118 @@
#!/usr/bin/env python3
"""
Test script for imbalance calculation logic
"""
import time
import logging
from datetime import datetime
from core.data_provider import DataProvider
# Set up logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
def test_imbalance_calculation():
"""Test the imbalance calculation logic with mock data"""
logger.info("Testing imbalance calculation logic...")
# Initialize data provider
dp = DataProvider()
# Create mock COB tick data
mock_ticks = []
current_time = time.time()
# Create 10 mock ticks with different imbalances
for i in range(10):
tick = {
'symbol': 'ETH/USDT',
'timestamp': current_time - (10 - i), # 10 seconds ago to now
'bids': [
[3800 + i, 100 + i * 10], # Price, Volume
[3799 + i, 50 + i * 5],
[3798 + i, 25 + i * 2]
],
'asks': [
[3801 + i, 80 + i * 8], # Price, Volume
[3802 + i, 40 + i * 4],
[3803 + i, 20 + i * 2]
],
'stats': {
'mid_price': 3800.5 + i,
'spread_bps': 2.5 + i * 0.1,
'imbalance': (i - 5) * 0.1 # Varying imbalance from -0.5 to +0.4
},
'source': 'mock'
}
mock_ticks.append(tick)
# Add mock ticks to the data provider
for tick in mock_ticks:
dp.cob_raw_ticks['ETH/USDT'].append(tick)
logger.info(f"Added {len(mock_ticks)} mock COB ticks")
# Test the aggregation function
logger.info("\n=== Testing COB Aggregation ===")
target_second = int(current_time - 5) # 5 seconds ago
# Manually call the aggregation function
dp._aggregate_cob_1s('ETH/USDT', target_second)
# Check the results
aggregated_data = list(dp.cob_1s_aggregated['ETH/USDT'])
if aggregated_data:
latest = aggregated_data[-1]
logger.info(f"Aggregated data created:")
logger.info(f" Timestamp: {latest.get('timestamp')}")
logger.info(f" Tick count: {latest.get('tick_count')}")
logger.info(f" Current imbalance: {latest.get('imbalance', 0):.4f}")
logger.info(f" Total volume: {latest.get('total_volume', 0):.2f}")
logger.info(f" Bid buckets: {len(latest.get('bid_buckets', {}))}")
logger.info(f" Ask buckets: {len(latest.get('ask_buckets', {}))}")
# Check multi-timeframe imbalances
logger.info(f" Imbalance 1s: {latest.get('imbalance_1s', 0):.4f}")
logger.info(f" Imbalance 5s: {latest.get('imbalance_5s', 0):.4f}")
logger.info(f" Imbalance 15s: {latest.get('imbalance_15s', 0):.4f}")
logger.info(f" Imbalance 60s: {latest.get('imbalance_60s', 0):.4f}")
else:
logger.warning("No aggregated data created")
# Test multiple aggregations to build history
logger.info("\n=== Testing Multi-timeframe Imbalances ===")
for i in range(1, 6):
target_second = int(current_time - 5 + i)
dp._aggregate_cob_1s('ETH/USDT', target_second)
# Check the final results
final_data = list(dp.cob_1s_aggregated['ETH/USDT'])
logger.info(f"Created {len(final_data)} aggregated records")
if final_data:
latest = final_data[-1]
logger.info(f"Final imbalance indicators:")
logger.info(f" 1s: {latest.get('imbalance_1s', 0):.4f}")
logger.info(f" 5s: {latest.get('imbalance_5s', 0):.4f}")
logger.info(f" 15s: {latest.get('imbalance_15s', 0):.4f}")
logger.info(f" 60s: {latest.get('imbalance_60s', 0):.4f}")
# Test the COB data quality function
logger.info("\n=== Testing COB Data Quality Function ===")
quality = dp.get_cob_data_quality()
eth_quality = quality.get('imbalance_indicators', {}).get('ETH/USDT', {})
if eth_quality:
logger.info("COB quality indicators:")
logger.info(f" Imbalance 1s: {eth_quality.get('imbalance_1s', 0):.4f}")
logger.info(f" Imbalance 5s: {eth_quality.get('imbalance_5s', 0):.4f}")
logger.info(f" Imbalance 15s: {eth_quality.get('imbalance_15s', 0):.4f}")
logger.info(f" Imbalance 60s: {eth_quality.get('imbalance_60s', 0):.4f}")
logger.info(f" Total volume: {eth_quality.get('total_volume', 0):.2f}")
logger.info(f" Bucket count: {eth_quality.get('bucket_count', 0)}")
logger.info("\n✅ Imbalance calculation test completed successfully!")
if __name__ == "__main__":
test_imbalance_calculation()

52
test_orchestrator_fix.py Normal file
View File

@ -0,0 +1,52 @@
#!/usr/bin/env python3
"""
Test script to verify orchestrator fix
"""
import logging
import os
# Fix OpenMP library conflicts
os.environ['KMP_DUPLICATE_LIB_OK'] = 'TRUE'
os.environ['OMP_NUM_THREADS'] = '4'
# Fix matplotlib backend
import matplotlib
matplotlib.use('Agg')
# Set up logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
def test_orchestrator():
"""Test orchestrator initialization"""
try:
logger.info("Testing orchestrator initialization...")
# Import required modules
from core.standardized_data_provider import StandardizedDataProvider
from core.orchestrator import TradingOrchestrator
logger.info("Imports successful")
# Create data provider
data_provider = StandardizedDataProvider()
logger.info("StandardizedDataProvider created")
# Create orchestrator
orchestrator = TradingOrchestrator(data_provider, enhanced_rl_training=True)
logger.info("TradingOrchestrator created successfully!")
# Test basic functionality
status = orchestrator.get_queue_status()
logger.info(f"Queue status: {status}")
logger.info("✅ Orchestrator test completed successfully!")
except Exception as e:
logger.error(f"❌ Orchestrator test failed: {e}")
import traceback
traceback.print_exc()
if __name__ == "__main__":
test_orchestrator()

153
test_timezone_handling.py Normal file
View File

@ -0,0 +1,153 @@
#!/usr/bin/env python3
"""
Test script to verify timezone handling in data provider
"""
import time
import logging
import pandas as pd
from datetime import datetime, timezone
from core.data_provider import DataProvider
# Set up logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
def test_timezone_handling():
"""Test timezone handling in data provider"""
logger.info("Testing timezone handling...")
# Initialize data provider
dp = DataProvider()
# Wait for initial data load
logger.info("Waiting for initial data load...")
time.sleep(15)
# Test 1: Check timezone info in cached data
logger.info("\n=== Test 1: Timezone Info in Cached Data ===")
for symbol in ['ETH/USDT', 'BTC/USDT']:
for timeframe in ['1s', '1m', '1h', '1d']:
if symbol in dp.cached_data and timeframe in dp.cached_data[symbol]:
df = dp.cached_data[symbol][timeframe]
if not df.empty:
# Check if index has timezone info
has_tz = df.index.tz is not None
tz_info = df.index.tz if has_tz else "No timezone"
# Get first and last timestamps
first_ts = df.index[0]
last_ts = df.index[-1]
logger.info(f"{symbol} {timeframe}:")
logger.info(f" Timezone: {tz_info}")
logger.info(f" First: {first_ts}")
logger.info(f" Last: {last_ts}")
# Check for gaps (only for timeframes with enough data)
if len(df) > 10:
# Calculate expected time difference
if timeframe == '1s':
expected_diff = pd.Timedelta(seconds=1)
elif timeframe == '1m':
expected_diff = pd.Timedelta(minutes=1)
elif timeframe == '1h':
expected_diff = pd.Timedelta(hours=1)
elif timeframe == '1d':
expected_diff = pd.Timedelta(days=1)
# Check for large gaps
time_diffs = df.index.to_series().diff()
large_gaps = time_diffs[time_diffs > expected_diff * 2]
if not large_gaps.empty:
logger.warning(f" Found {len(large_gaps)} large gaps:")
for gap_time, gap_size in large_gaps.head(3).items():
logger.warning(f" Gap at {gap_time}: {gap_size}")
else:
logger.info(f" No significant gaps found")
else:
logger.info(f"{symbol} {timeframe}: No data")
# Test 2: Compare with current UTC time
logger.info("\n=== Test 2: Compare with Current UTC Time ===")
current_utc = datetime.now(timezone.utc)
logger.info(f"Current UTC time: {current_utc}")
for symbol in ['ETH/USDT', 'BTC/USDT']:
# Get latest 1m data
if symbol in dp.cached_data and '1m' in dp.cached_data[symbol]:
df = dp.cached_data[symbol]['1m']
if not df.empty:
latest_ts = df.index[-1]
# Convert to UTC if it has timezone info
if latest_ts.tz is not None:
latest_utc = latest_ts.tz_convert('UTC')
else:
# Assume it's already UTC if no timezone
latest_utc = latest_ts.replace(tzinfo=timezone.utc)
time_diff = current_utc - latest_utc
logger.info(f"{symbol} latest data:")
logger.info(f" Timestamp: {latest_ts}")
logger.info(f" UTC: {latest_utc}")
logger.info(f" Age: {time_diff}")
# Check if data is reasonably fresh (within 1 hour)
if time_diff.total_seconds() < 3600:
logger.info(f" ✅ Data is fresh")
else:
logger.warning(f" ⚠️ Data is stale (>{time_diff})")
# Test 3: Check data continuity
logger.info("\n=== Test 3: Data Continuity Check ===")
for symbol in ['ETH/USDT', 'BTC/USDT']:
if symbol in dp.cached_data and '1h' in dp.cached_data[symbol]:
df = dp.cached_data[symbol]['1h']
if len(df) > 24: # At least 24 hours of data
# Get last 24 hours
recent_df = df.tail(24)
# Check for 3-hour gaps (the reported issue)
time_diffs = recent_df.index.to_series().diff()
three_hour_gaps = time_diffs[time_diffs >= pd.Timedelta(hours=3)]
logger.info(f"{symbol} 1h data (last 24 candles):")
logger.info(f" Time range: {recent_df.index[0]} to {recent_df.index[-1]}")
if not three_hour_gaps.empty:
logger.warning(f" ❌ Found {len(three_hour_gaps)} gaps >= 3 hours:")
for gap_time, gap_size in three_hour_gaps.items():
logger.warning(f" {gap_time}: {gap_size}")
else:
logger.info(f" ✅ No 3+ hour gaps found")
# Show time differences
logger.info(f" Time differences (last 5):")
for i, (ts, diff) in enumerate(time_diffs.tail(5).items()):
if pd.notna(diff):
logger.info(f" {ts}: {diff}")
# Test 4: Manual timezone conversion test
logger.info("\n=== Test 4: Manual Timezone Conversion Test ===")
# Create test timestamps
utc_now = datetime.now(timezone.utc)
local_now = datetime.now()
logger.info(f"UTC now: {utc_now}")
logger.info(f"Local now: {local_now}")
logger.info(f"Difference: {utc_now - local_now.replace(tzinfo=timezone.utc)}")
# Test pandas timezone handling
test_ts = pd.Timestamp.now(tz='UTC')
logger.info(f"Pandas UTC timestamp: {test_ts}")
# Clean shutdown
logger.info("\n=== Shutting Down ===")
dp.stop_automatic_data_maintenance()
logger.info("Timezone handling test completed")
if __name__ == "__main__":
test_timezone_handling()

111
test_websocket_cob_data.py Normal file
View File

@ -0,0 +1,111 @@
#!/usr/bin/env python3
"""
Test script to check if we're getting real COB data from WebSocket
"""
import time
import logging
from core.data_provider import DataProvider
# Set up logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
def test_websocket_cob_data():
"""Test if we're getting real COB data from WebSocket"""
logger.info("Testing WebSocket COB data reception...")
# Initialize data provider
dp = DataProvider()
# Wait for WebSocket connections
logger.info("Waiting for WebSocket connections...")
time.sleep(15)
# Check WebSocket status
logger.info("\n=== WebSocket Status ===")
try:
if hasattr(dp, 'enhanced_cob_websocket') and dp.enhanced_cob_websocket:
status = dp.enhanced_cob_websocket.get_status_summary()
logger.info(f"WebSocket status: {status}")
else:
logger.warning("Enhanced COB WebSocket not available")
except Exception as e:
logger.error(f"Error getting WebSocket status: {e}")
# Check if we have any COB WebSocket data
logger.info("\n=== COB WebSocket Data Check ===")
if hasattr(dp, 'cob_websocket_data'):
for symbol in ['ETH/USDT', 'BTC/USDT']:
if symbol in dp.cob_websocket_data:
data = dp.cob_websocket_data[symbol]
logger.info(f"{symbol}: {type(data)} - {len(str(data))} chars")
if isinstance(data, dict):
logger.info(f" Keys: {list(data.keys())}")
if 'bids' in data:
logger.info(f" Bids: {len(data['bids'])} levels")
if 'asks' in data:
logger.info(f" Asks: {len(data['asks'])} levels")
else:
logger.info(f"{symbol}: No WebSocket data")
else:
logger.warning("No cob_websocket_data attribute found")
# Check raw COB ticks
logger.info("\n=== Raw COB Ticks ===")
for symbol in ['ETH/USDT', 'BTC/USDT']:
if hasattr(dp, 'cob_raw_ticks') and symbol in dp.cob_raw_ticks:
raw_ticks = list(dp.cob_raw_ticks[symbol])
logger.info(f"{symbol}: {len(raw_ticks)} raw ticks")
if raw_ticks:
latest = raw_ticks[-1]
logger.info(f" Latest tick keys: {list(latest.keys())}")
if 'timestamp' in latest:
logger.info(f" Latest timestamp: {latest['timestamp']}")
else:
logger.info(f"{symbol}: No raw ticks")
# Monitor for 30 seconds to see if data comes in
logger.info("\n=== Monitoring for 30 seconds ===")
initial_counts = {}
for symbol in ['ETH/USDT', 'BTC/USDT']:
if hasattr(dp, 'cob_raw_ticks') and symbol in dp.cob_raw_ticks:
initial_counts[symbol] = len(dp.cob_raw_ticks[symbol])
else:
initial_counts[symbol] = 0
time.sleep(30)
logger.info("After 30 seconds:")
for symbol in ['ETH/USDT', 'BTC/USDT']:
if hasattr(dp, 'cob_raw_ticks') and symbol in dp.cob_raw_ticks:
current_count = len(dp.cob_raw_ticks[symbol])
new_ticks = current_count - initial_counts[symbol]
logger.info(f"{symbol}: +{new_ticks} new ticks (total: {current_count})")
else:
logger.info(f"{symbol}: No raw ticks available")
# Check if Enhanced WebSocket has latest data
logger.info("\n=== Enhanced WebSocket Latest Data ===")
try:
if hasattr(dp, 'enhanced_cob_websocket') and dp.enhanced_cob_websocket:
for symbol in ['ETH/USDT', 'BTC/USDT']:
if hasattr(dp.enhanced_cob_websocket, 'latest_cob_data'):
latest_data = dp.enhanced_cob_websocket.latest_cob_data.get(symbol)
if latest_data:
logger.info(f"{symbol}: Latest WebSocket data available")
logger.info(f" Keys: {list(latest_data.keys())}")
if 'bids' in latest_data and 'asks' in latest_data:
logger.info(f" Bids: {len(latest_data['bids'])}, Asks: {len(latest_data['asks'])}")
else:
logger.info(f"{symbol}: No latest WebSocket data")
except Exception as e:
logger.error(f"Error checking Enhanced WebSocket data: {e}")
# Clean shutdown
logger.info("\n=== Shutting Down ===")
dp.stop_automatic_data_maintenance()
logger.info("WebSocket COB data test completed")
if __name__ == "__main__":
test_websocket_cob_data()