Merge commit 'd49a473ed6f4aef55bfdd47d6370e53582be6b7b' into cleanup
This commit is contained in:
Dobromir Popov
349
docs/ENHANCED_REWARD_SYSTEM.md
Normal file
349
docs/ENHANCED_REWARD_SYSTEM.md
Normal file
@@ -0,0 +1,349 @@
|
||||
# Enhanced Reward System for Reinforcement Learning Training
|
||||
|
||||
## Overview
|
||||
|
||||
This document describes the implementation of an enhanced reward system for your reinforcement learning trading models. The system uses **mean squared error (MSE) between predictions and empirical outcomes** as the primary reward mechanism, with support for multiple timeframes and comprehensive accuracy tracking.
|
||||
|
||||
## Key Features
|
||||
|
||||
### ✅ MSE-Based Reward Calculation
|
||||
- Uses mean squared difference between predicted and actual prices
|
||||
- Exponential decay function heavily penalizes large prediction errors
|
||||
- Direction accuracy bonus/penalty system
|
||||
- Confidence-weighted final rewards
|
||||
|
||||
### ✅ Multi-Timeframe Support
|
||||
- Separate tracking for **1s, 1m, 1h, 1d** timeframes
|
||||
- Independent accuracy metrics for each timeframe
|
||||
- Timeframe-specific evaluation timeouts
|
||||
- Models know which timeframe they're predicting on
|
||||
|
||||
### ✅ Prediction History Tracking
|
||||
- Maintains last **6 predictions per timeframe** per symbol
|
||||
- Comprehensive prediction records with outcomes
|
||||
- Historical accuracy analysis
|
||||
- Memory-efficient with automatic cleanup
|
||||
|
||||
### ✅ Real-Time Training
|
||||
- Training triggered at each inference when outcomes are available
|
||||
- Separate training batches for each model and timeframe
|
||||
- Automatic evaluation of predictions after appropriate timeouts
|
||||
- Integration with existing RL training infrastructure
|
||||
|
||||
### ✅ Enhanced Inference Scheduling
|
||||
- **Continuous inference** every 1-5 seconds on primary timeframe
|
||||
- **Hourly multi-timeframe inference** (4 predictions per hour - one for each timeframe)
|
||||
- Timeframe-aware inference context
|
||||
- Proper scheduling and coordination
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[Market Data] --> B[Timeframe Inference Coordinator]
|
||||
B --> C[Model Inference]
|
||||
C --> D[Enhanced Reward Calculator]
|
||||
D --> E[Prediction Tracking]
|
||||
E --> F[Outcome Evaluation]
|
||||
F --> G[MSE Reward Calculation]
|
||||
G --> H[Enhanced RL Training Adapter]
|
||||
H --> I[Model Training]
|
||||
I --> J[Performance Monitoring]
|
||||
```
|
||||
|
||||
## Core Components
|
||||
|
||||
### 1. EnhancedRewardCalculator (`core/enhanced_reward_calculator.py`)
|
||||
|
||||
**Purpose**: Central reward calculation engine using MSE methodology
|
||||
|
||||
**Key Methods**:
|
||||
- `add_prediction()` - Track new predictions
|
||||
- `evaluate_predictions()` - Calculate rewards when outcomes available
|
||||
- `get_accuracy_summary()` - Comprehensive accuracy metrics
|
||||
- `get_training_data()` - Extract training samples for models
|
||||
|
||||
**Reward Formula**:
|
||||
```python
|
||||
# MSE calculation
|
||||
price_error = actual_price - predicted_price
|
||||
mse = price_error ** 2
|
||||
|
||||
# Normalize to reasonable scale
|
||||
max_mse = (current_price * 0.1) ** 2 # 10% as max expected error
|
||||
normalized_mse = min(mse / max_mse, 1.0)
|
||||
|
||||
# Exponential decay (heavily penalize large errors)
|
||||
mse_reward = exp(-5 * normalized_mse) # Range: [exp(-5), 1]
|
||||
|
||||
# Direction bonus/penalty
|
||||
direction_bonus = 0.5 if direction_correct else -0.5
|
||||
|
||||
# Final reward (confidence weighted)
|
||||
final_reward = (mse_reward + direction_bonus) * confidence
|
||||
```
|
||||
|
||||
### 2. TimeframeInferenceCoordinator (`core/timeframe_inference_coordinator.py`)
|
||||
|
||||
**Purpose**: Coordinates timeframe-aware model inference with proper scheduling
|
||||
|
||||
**Key Features**:
|
||||
- **Continuous inference loop** for each symbol (every 5 seconds)
|
||||
- **Hourly multi-timeframe scheduler** (4 predictions per hour)
|
||||
- **Inference context management** (models know target timeframe)
|
||||
- **Automatic reward evaluation** and training triggers
|
||||
|
||||
**Scheduling**:
|
||||
- **Every 5 seconds**: Inference on primary timeframe (1s)
|
||||
- **Every hour**: One inference for each timeframe (1s, 1m, 1h, 1d)
|
||||
- **Evaluation timeouts**: 5s for 1s predictions, 60s for 1m, 300s for 1h, 900s for 1d
|
||||
|
||||
### 3. EnhancedRLTrainingAdapter (`core/enhanced_rl_training_adapter.py`)
|
||||
|
||||
**Purpose**: Bridge between new reward system and existing RL training infrastructure
|
||||
|
||||
**Key Features**:
|
||||
- **Model inference wrappers** for DQN, COB RL, and CNN models
|
||||
- **Training batch creation** from prediction records and rewards
|
||||
- **Real-time training triggers** based on evaluation results
|
||||
- **Backward compatibility** with existing training systems
|
||||
|
||||
### 4. EnhancedRewardSystemIntegration (`core/enhanced_reward_system_integration.py`)
|
||||
|
||||
**Purpose**: Simple integration point for existing systems
|
||||
|
||||
**Key Features**:
|
||||
- **One-line integration** with existing TradingOrchestrator
|
||||
- **Helper functions** for easy prediction tracking
|
||||
- **Comprehensive monitoring** and statistics
|
||||
- **Minimal code changes** required
|
||||
|
||||
## Integration Guide
|
||||
|
||||
### Step 1: Import Required Components
|
||||
|
||||
Add to your `orchestrator.py`:
|
||||
|
||||
```python
|
||||
from core.enhanced_reward_system_integration import (
|
||||
integrate_enhanced_rewards,
|
||||
add_prediction_to_enhanced_rewards
|
||||
)
|
||||
```
|
||||
|
||||
### Step 2: Initialize in TradingOrchestrator
|
||||
|
||||
In your `TradingOrchestrator.__init__()`:
|
||||
|
||||
```python
|
||||
# Add this line after existing initialization
|
||||
integrate_enhanced_rewards(self, symbols=['ETH/USDT', 'BTC/USDT'])
|
||||
```
|
||||
|
||||
### Step 3: Start the System
|
||||
|
||||
In your `TradingOrchestrator.run()` method:
|
||||
|
||||
```python
|
||||
# Add this line after initialization
|
||||
await self.enhanced_reward_system.start_integration()
|
||||
```
|
||||
|
||||
### Step 4: Track Predictions
|
||||
|
||||
In your model inference methods (CNN, DQN, COB RL):
|
||||
|
||||
```python
|
||||
# Example in CNN inference
|
||||
prediction_id = add_prediction_to_enhanced_rewards(
|
||||
self, # orchestrator instance
|
||||
symbol, # 'ETH/USDT'
|
||||
timeframe, # '1s', '1m', '1h', '1d'
|
||||
predicted_price, # model's price prediction
|
||||
direction, # -1 (down), 0 (neutral), 1 (up)
|
||||
confidence, # 0.0 to 1.0
|
||||
current_price, # current market price
|
||||
'enhanced_cnn' # model name
|
||||
)
|
||||
```
|
||||
|
||||
### Step 5: Monitor Performance
|
||||
|
||||
```python
|
||||
# Get comprehensive statistics
|
||||
stats = self.enhanced_reward_system.get_integration_statistics()
|
||||
accuracy = self.enhanced_reward_system.get_model_accuracy()
|
||||
|
||||
# Force evaluation for testing
|
||||
self.enhanced_reward_system.force_evaluation_and_training('ETH/USDT', '1s')
|
||||
```
|
||||
|
||||
## Usage Example
|
||||
|
||||
See `examples/enhanced_reward_system_example.py` for a complete demonstration.
|
||||
|
||||
```bash
|
||||
python examples/enhanced_reward_system_example.py
|
||||
```
|
||||
|
||||
## Performance Benefits
|
||||
|
||||
### 🎯 Better Accuracy Measurement
|
||||
- **MSE rewards** provide nuanced feedback vs. simple directional accuracy
|
||||
- **Price prediction accuracy** measured alongside direction accuracy
|
||||
- **Confidence-weighted rewards** encourage well-calibrated predictions
|
||||
|
||||
### 📊 Multi-Timeframe Intelligence
|
||||
- **Separate tracking** prevents timeframe confusion
|
||||
- **Timeframe-specific evaluation** accounts for different market dynamics
|
||||
- **Comprehensive accuracy picture** across all prediction horizons
|
||||
|
||||
### ⚡ Real-Time Learning
|
||||
- **Immediate training** when prediction outcomes available
|
||||
- **No batch delays** - models learn from every prediction
|
||||
- **Adaptive training frequency** based on prediction evaluation
|
||||
|
||||
### 🔄 Enhanced Inference Scheduling
|
||||
- **Optimal prediction frequency** balances real-time response with computational efficiency
|
||||
- **Hourly multi-timeframe predictions** provide comprehensive market coverage
|
||||
- **Context-aware models** make better predictions knowing their target timeframe
|
||||
|
||||
## Configuration
|
||||
|
||||
### Evaluation Timeouts (Configurable in EnhancedRewardCalculator)
|
||||
|
||||
```python
|
||||
evaluation_timeouts = {
|
||||
TimeFrame.SECONDS_1: 5, # Evaluate 1s predictions after 5 seconds
|
||||
TimeFrame.MINUTES_1: 60, # Evaluate 1m predictions after 1 minute
|
||||
TimeFrame.HOURS_1: 300, # Evaluate 1h predictions after 5 minutes
|
||||
TimeFrame.DAYS_1: 900 # Evaluate 1d predictions after 15 minutes
|
||||
}
|
||||
```
|
||||
|
||||
### Inference Scheduling (Configurable in TimeframeInferenceCoordinator)
|
||||
|
||||
```python
|
||||
schedule = InferenceSchedule(
|
||||
continuous_interval_seconds=5.0, # Continuous inference every 5 seconds
|
||||
hourly_timeframes=[TimeFrame.SECONDS_1, TimeFrame.MINUTES_1,
|
||||
TimeFrame.HOURS_1, TimeFrame.DAYS_1]
|
||||
)
|
||||
```
|
||||
|
||||
### Training Configuration (Configurable in EnhancedRLTrainingAdapter)
|
||||
|
||||
```python
|
||||
min_batch_size = 8 # Minimum samples for training
|
||||
max_batch_size = 64 # Maximum samples per training batch
|
||||
training_interval_seconds = 5.0 # Training check frequency
|
||||
```
|
||||
|
||||
## Monitoring and Statistics
|
||||
|
||||
### Integration Statistics
|
||||
|
||||
```python
|
||||
stats = enhanced_reward_system.get_integration_statistics()
|
||||
```
|
||||
|
||||
Returns:
|
||||
- System running status
|
||||
- Total predictions tracked
|
||||
- Component status
|
||||
- Inference and training statistics
|
||||
- Performance metrics
|
||||
|
||||
### Model Accuracy
|
||||
|
||||
```python
|
||||
accuracy = enhanced_reward_system.get_model_accuracy()
|
||||
```
|
||||
|
||||
Returns for each symbol and timeframe:
|
||||
- Total predictions made
|
||||
- Direction accuracy percentage
|
||||
- Average MSE
|
||||
- Recent prediction count
|
||||
|
||||
### Real-Time Monitoring
|
||||
|
||||
The system provides comprehensive logging at different levels:
|
||||
- `INFO`: Major system events, training results
|
||||
- `DEBUG`: Detailed prediction tracking, reward calculations
|
||||
- `ERROR`: System errors and recovery actions
|
||||
|
||||
## Backward Compatibility
|
||||
|
||||
The enhanced reward system is designed to be **fully backward compatible**:
|
||||
|
||||
✅ **Existing models continue to work** without modification
|
||||
✅ **Existing training systems** remain functional
|
||||
✅ **Existing reward calculations** can run in parallel
|
||||
✅ **Gradual migration** - enable for specific models incrementally
|
||||
|
||||
## Testing and Validation
|
||||
|
||||
### Force Evaluation for Testing
|
||||
|
||||
```python
|
||||
# Force immediate evaluation of all predictions
|
||||
enhanced_reward_system.force_evaluation_and_training()
|
||||
|
||||
# Force evaluation for specific symbol/timeframe
|
||||
enhanced_reward_system.force_evaluation_and_training('ETH/USDT', '1s')
|
||||
```
|
||||
|
||||
### Manual Prediction Addition
|
||||
|
||||
```python
|
||||
# Add predictions manually for testing
|
||||
prediction_id = enhanced_reward_system.add_prediction_manually(
|
||||
symbol='ETH/USDT',
|
||||
timeframe_str='1s',
|
||||
predicted_price=3150.50,
|
||||
predicted_direction=1,
|
||||
confidence=0.85,
|
||||
current_price=3150.00,
|
||||
model_name='test_model'
|
||||
)
|
||||
```
|
||||
|
||||
## Memory Management
|
||||
|
||||
The system includes automatic memory management:
|
||||
|
||||
- **Automatic prediction cleanup** (configurable retention period)
|
||||
- **Circular buffers** for prediction history (max 100 per timeframe)
|
||||
- **Price cache management** (max 1000 price points per symbol)
|
||||
- **Efficient storage** using deques and compressed data structures
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
The architecture supports easy extension for:
|
||||
|
||||
1. **Additional timeframes** (30s, 5m, 15m, etc.)
|
||||
2. **Custom reward functions** (Sharpe ratio, maximum drawdown, etc.)
|
||||
3. **Multi-symbol correlation** rewards
|
||||
4. **Advanced statistical metrics** (Sortino ratio, Calmar ratio)
|
||||
5. **Model ensemble** reward aggregation
|
||||
6. **A/B testing** framework for reward functions
|
||||
|
||||
## Conclusion
|
||||
|
||||
The Enhanced Reward System provides a comprehensive foundation for improving RL model training through:
|
||||
|
||||
- **Precise MSE-based rewards** that accurately measure prediction quality
|
||||
- **Multi-timeframe intelligence** that prevents confusion between different prediction horizons
|
||||
- **Real-time learning** that maximizes training opportunities
|
||||
- **Easy integration** that requires minimal changes to existing code
|
||||
- **Comprehensive monitoring** that provides insights into model performance
|
||||
|
||||
This system addresses the specific requirements you outlined:
|
||||
✅ MSE-based accuracy calculation
|
||||
✅ Training at each inference using last prediction vs. current outcome
|
||||
✅ Separate accuracy tracking for up to 6 last predictions per timeframe
|
||||
✅ Models know which timeframe they're predicting on
|
||||
✅ Hourly multi-timeframe inference (4 predictions per hour)
|
||||
✅ Integration with existing 1-5 second inference frequency
|
||||
|
||||
0
docs/cache_corruption_fix.md
Normal file
0
docs/cache_corruption_fix.md
Normal file
104
docs/exchanges/bybit/README.md
Normal file
104
docs/exchanges/bybit/README.md
Normal file
@@ -0,0 +1,104 @@
|
||||
# Bybit Exchange Integration Documentation
|
||||
|
||||
## Overview
|
||||
This documentation covers the integration of Bybit exchange using the official pybit Python library.
|
||||
|
||||
**Library:** [pybit](https://github.com/bybit-exchange/pybit)
|
||||
**Version:** 5.11.0 (Latest as of 2025-01-26)
|
||||
**Official Repository:** https://github.com/bybit-exchange/pybit
|
||||
|
||||
## Installation
|
||||
```bash
|
||||
pip install pybit
|
||||
```
|
||||
|
||||
## Requirements
|
||||
- Python 3.9.1 or higher
|
||||
- API credentials (BYBIT_API_KEY and BYBIT_API_SECRET)
|
||||
|
||||
## Basic Usage
|
||||
|
||||
### HTTP Session Creation
|
||||
```python
|
||||
from pybit.unified_trading import HTTP
|
||||
|
||||
# Create HTTP session
|
||||
session = HTTP(
|
||||
testnet=False, # Set to True for testnet
|
||||
api_key="your_api_key",
|
||||
api_secret="your_api_secret",
|
||||
)
|
||||
```
|
||||
|
||||
### Common Operations
|
||||
|
||||
#### Get Orderbook
|
||||
```python
|
||||
# Get orderbook for BTCUSDT perpetual
|
||||
orderbook = session.get_orderbook(category="linear", symbol="BTCUSDT")
|
||||
```
|
||||
|
||||
#### Place Order
|
||||
```python
|
||||
# Place a single order
|
||||
order = session.place_order(
|
||||
category="linear",
|
||||
symbol="BTCUSDT",
|
||||
side="Buy",
|
||||
orderType="Limit",
|
||||
qty="0.001",
|
||||
price="50000"
|
||||
)
|
||||
```
|
||||
|
||||
#### Batch Orders (USDC Options only)
|
||||
```python
|
||||
# Create multiple orders (USDC Options support only)
|
||||
payload = {"category": "option"}
|
||||
orders = [{
|
||||
"symbol": "BTC-30JUN23-20000-C",
|
||||
"side": "Buy",
|
||||
"orderType": "Limit",
|
||||
"qty": "0.1",
|
||||
"price": str(15000 + i * 500),
|
||||
} for i in range(5)]
|
||||
|
||||
payload["request"] = orders
|
||||
session.place_batch_order(payload)
|
||||
```
|
||||
|
||||
## Categories
|
||||
- **linear**: USDT Perpetuals (BTCUSDT, ETHUSDT, etc.)
|
||||
- **inverse**: Inverse Perpetuals
|
||||
- **option**: USDC Options
|
||||
- **spot**: Spot trading
|
||||
|
||||
## Key Features
|
||||
- Official Bybit library maintained by Bybit employees
|
||||
- Lightweight with minimal external dependencies
|
||||
- Support for both HTTP and WebSocket APIs
|
||||
- Active development and quick API updates
|
||||
- Built-in testnet support
|
||||
|
||||
## Dependencies
|
||||
- `requests` - HTTP API calls
|
||||
- `websocket-client` - WebSocket connections
|
||||
- Built-in Python modules
|
||||
|
||||
## Trading Pairs
|
||||
- BTC/USDT perpetuals
|
||||
- ETH/USDT perpetuals
|
||||
- Various altcoin perpetuals
|
||||
- Options contracts
|
||||
- Spot markets
|
||||
|
||||
## Environment Variables
|
||||
- `BYBIT_API_KEY` - Your Bybit API key
|
||||
- `BYBIT_API_SECRET` - Your Bybit API secret
|
||||
|
||||
## Integration Notes
|
||||
- Unified trading interface for all Bybit products
|
||||
- Consistent API structure across different categories
|
||||
- Comprehensive error handling
|
||||
- Rate limiting compliance
|
||||
- Active community support via Telegram and Discord
|
||||
311
docs/fifo_queue_system.md
Normal file
311
docs/fifo_queue_system.md
Normal file
@@ -0,0 +1,311 @@
|
||||
# FIFO Queue System for Data Management
|
||||
|
||||
## Problem
|
||||
|
||||
The CNN model was constantly rebuilding its network architecture at runtime due to inconsistent input dimensions:
|
||||
|
||||
```
|
||||
2025-07-25 23:53:33,053 - NN.models.enhanced_cnn - INFO - Rebuilding network for new feature dimension: 300 (was 7850)
|
||||
2025-07-25 23:53:33,969 - NN.models.enhanced_cnn - INFO - Rebuilding network for new feature dimension: 7850 (was 300)
|
||||
```
|
||||
|
||||
**Root Causes**:
|
||||
1. **Inconsistent data availability** - Different refresh rates for various data types
|
||||
2. **Direct data provider calls** - Models getting data at different times with varying completeness
|
||||
3. **No data buffering** - Missing data causing feature vector size variations
|
||||
4. **Race conditions** - Multiple models accessing data provider simultaneously
|
||||
|
||||
## Solution: FIFO Queue System
|
||||
|
||||
### 1. **FIFO Data Queues** (`core/orchestrator.py`)
|
||||
|
||||
**Centralized data buffering**:
|
||||
```python
|
||||
self.data_queues = {
|
||||
'ohlcv_1s': {symbol: deque(maxlen=500) for symbol in [self.symbol] + self.ref_symbols},
|
||||
'ohlcv_1m': {symbol: deque(maxlen=300) for symbol in [self.symbol] + self.ref_symbols},
|
||||
'ohlcv_1h': {symbol: deque(maxlen=300) for symbol in [self.symbol] + self.ref_symbols},
|
||||
'ohlcv_1d': {symbol: deque(maxlen=300) for symbol in [self.symbol] + self.ref_symbols},
|
||||
'technical_indicators': {symbol: deque(maxlen=100) for symbol in [self.symbol] + self.ref_symbols},
|
||||
'cob_data': {symbol: deque(maxlen=50) for symbol in [self.symbol]},
|
||||
'model_predictions': {symbol: deque(maxlen=20) for symbol in [self.symbol]}
|
||||
}
|
||||
```
|
||||
|
||||
**Thread-safe operations**:
|
||||
```python
|
||||
self.data_queue_locks = {
|
||||
data_type: {symbol: threading.Lock() for symbol in queue_dict.keys()}
|
||||
for data_type, queue_dict in self.data_queues.items()
|
||||
}
|
||||
```
|
||||
|
||||
### 2. **Queue Management Methods**
|
||||
|
||||
**Update queues**:
|
||||
```python
|
||||
def update_data_queue(self, data_type: str, symbol: str, data: Any) -> bool:
|
||||
"""Thread-safe queue update with new data"""
|
||||
with self.data_queue_locks[data_type][symbol]:
|
||||
self.data_queues[data_type][symbol].append(data)
|
||||
```
|
||||
|
||||
**Retrieve data**:
|
||||
```python
|
||||
def get_queue_data(self, data_type: str, symbol: str, max_items: int = None) -> List[Any]:
|
||||
"""Get all data from FIFO queue with optional limit"""
|
||||
with self.data_queue_locks[data_type][symbol]:
|
||||
queue = self.data_queues[data_type][symbol]
|
||||
return list(queue)[-max_items:] if max_items else list(queue)
|
||||
```
|
||||
|
||||
**Check data availability**:
|
||||
```python
|
||||
def ensure_minimum_data(self, data_type: str, symbol: str, min_count: int) -> bool:
|
||||
"""Verify queue has minimum required data"""
|
||||
with self.data_queue_locks[data_type][symbol]:
|
||||
return len(self.data_queues[data_type][symbol]) >= min_count
|
||||
```
|
||||
|
||||
### 3. **Consistent BaseDataInput Building**
|
||||
|
||||
**Fixed-size data construction**:
|
||||
```python
|
||||
def build_base_data_input(self, symbol: str) -> Optional[BaseDataInput]:
|
||||
"""Build BaseDataInput from FIFO queues with consistent data"""
|
||||
|
||||
# Check minimum data requirements
|
||||
min_requirements = {
|
||||
'ohlcv_1s': 100,
|
||||
'ohlcv_1m': 50,
|
||||
'ohlcv_1h': 20,
|
||||
'ohlcv_1d': 10
|
||||
}
|
||||
|
||||
# Verify minimum data availability
|
||||
for data_type, min_count in min_requirements.items():
|
||||
if not self.ensure_minimum_data(data_type, symbol, min_count):
|
||||
return None
|
||||
|
||||
# Build with consistent data from queues
|
||||
return BaseDataInput(
|
||||
symbol=symbol,
|
||||
timestamp=datetime.now(),
|
||||
ohlcv_1s=self.get_queue_data('ohlcv_1s', symbol, 300),
|
||||
ohlcv_1m=self.get_queue_data('ohlcv_1m', symbol, 300),
|
||||
ohlcv_1h=self.get_queue_data('ohlcv_1h', symbol, 300),
|
||||
ohlcv_1d=self.get_queue_data('ohlcv_1d', symbol, 300),
|
||||
btc_ohlcv_1s=self.get_queue_data('ohlcv_1s', 'BTC/USDT', 300),
|
||||
technical_indicators=self._get_latest_indicators(symbol),
|
||||
cob_data=self._get_latest_cob_data(symbol),
|
||||
last_predictions=self._get_recent_model_predictions(symbol)
|
||||
)
|
||||
```
|
||||
|
||||
### 4. **Data Integration System**
|
||||
|
||||
**Automatic queue population**:
|
||||
```python
|
||||
def _start_data_polling_thread(self):
|
||||
"""Background thread to poll data and populate queues"""
|
||||
def data_polling_worker():
|
||||
while self.running:
|
||||
# Poll OHLCV data for all symbols and timeframes
|
||||
for symbol in [self.symbol] + self.ref_symbols:
|
||||
for timeframe in ['1s', '1m', '1h', '1d']:
|
||||
data = self.data_provider.get_latest_ohlcv(symbol, timeframe, limit=1)
|
||||
if data and len(data) > 0:
|
||||
self.update_data_queue(f'ohlcv_{timeframe}', symbol, data[-1])
|
||||
|
||||
# Poll technical indicators and COB data
|
||||
# ... (similar polling for other data types)
|
||||
|
||||
time.sleep(1) # Poll every second
|
||||
```
|
||||
|
||||
### 5. **Fixed Feature Vector Size** (`core/data_models.py`)
|
||||
|
||||
**Guaranteed consistent size**:
|
||||
```python
|
||||
def get_feature_vector(self) -> np.ndarray:
|
||||
"""Convert BaseDataInput to FIXED SIZE standardized feature vector (7850 features)"""
|
||||
FIXED_FEATURE_SIZE = 7850
|
||||
features = []
|
||||
|
||||
# OHLCV features (6000 features: 300 frames x 4 timeframes x 5 features)
|
||||
for ohlcv_list in [self.ohlcv_1s, self.ohlcv_1m, self.ohlcv_1h, self.ohlcv_1d]:
|
||||
# Ensure exactly 300 frames by padding or truncating
|
||||
ohlcv_frames = ohlcv_list[-300:] if len(ohlcv_list) >= 300 else ohlcv_list
|
||||
while len(ohlcv_frames) < 300:
|
||||
dummy_bar = OHLCVBar(...) # Pad with zeros
|
||||
ohlcv_frames.insert(0, dummy_bar)
|
||||
|
||||
for bar in ohlcv_frames:
|
||||
features.extend([bar.open, bar.high, bar.low, bar.close, bar.volume])
|
||||
|
||||
# BTC OHLCV features (1500 features: 300 frames x 5 features)
|
||||
# COB features (200 features: fixed allocation)
|
||||
# Technical indicators (100 features: fixed allocation)
|
||||
# Model predictions (50 features: fixed allocation)
|
||||
|
||||
# CRITICAL: Ensure EXACTLY the fixed feature size
|
||||
assert len(features) == FIXED_FEATURE_SIZE
|
||||
return np.array(features, dtype=np.float32)
|
||||
```
|
||||
|
||||
### 6. **Enhanced CNN Protection** (`NN/models/enhanced_cnn.py`)
|
||||
|
||||
**No runtime rebuilding**:
|
||||
```python
|
||||
def _check_rebuild_network(self, features):
|
||||
"""DEPRECATED: Network should have fixed architecture - no runtime rebuilding"""
|
||||
if features != self.feature_dim:
|
||||
logger.error(f"CRITICAL: Input feature dimension mismatch! Expected {self.feature_dim}, got {features}")
|
||||
logger.error("This indicates a bug in data preprocessing - input should be fixed size!")
|
||||
raise ValueError(f"Input dimension mismatch: expected {self.feature_dim}, got {features}")
|
||||
return False
|
||||
```
|
||||
|
||||
## Benefits
|
||||
|
||||
### 1. **Consistent Data Flow**
|
||||
- **Before**: Models got different data depending on timing and availability
|
||||
- **After**: All models get consistent, complete data from FIFO queues
|
||||
|
||||
### 2. **No Network Rebuilding**
|
||||
- **Before**: CNN rebuilt architecture when input size changed (300 ↔ 7850)
|
||||
- **After**: Fixed 7850-feature input size, no runtime architecture changes
|
||||
|
||||
### 3. **Thread Safety**
|
||||
- **Before**: Race conditions when multiple models accessed data provider
|
||||
- **After**: Thread-safe queue operations with proper locking
|
||||
|
||||
### 4. **Data Availability Guarantee**
|
||||
- **Before**: Models might get incomplete data or fail due to missing data
|
||||
- **After**: Minimum data requirements checked before model inference
|
||||
|
||||
### 5. **Performance Improvement**
|
||||
- **Before**: Models waited for data provider calls, potential blocking
|
||||
- **After**: Instant data access from in-memory queues
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Data Provider → FIFO Queues → BaseDataInput → Models
|
||||
↓ ↓ ↓ ↓
|
||||
Real-time Thread-safe Fixed-size Stable
|
||||
Updates Buffering Features Architecture
|
||||
```
|
||||
|
||||
### Data Flow:
|
||||
1. **Data Provider** continuously fetches market data
|
||||
2. **Background Thread** polls data provider and updates FIFO queues
|
||||
3. **FIFO Queues** maintain rolling buffers of recent data
|
||||
4. **BaseDataInput Builder** constructs consistent input from queues
|
||||
5. **Models** receive fixed-size, complete data for inference
|
||||
|
||||
### Queue Sizes:
|
||||
- **OHLCV 1s**: 500 bars (8+ minutes of data)
|
||||
- **OHLCV 1m**: 300 bars (5 hours of data)
|
||||
- **OHLCV 1h**: 300 bars (12+ days of data)
|
||||
- **OHLCV 1d**: 300 bars (10+ months of data)
|
||||
- **Technical Indicators**: 100 latest values
|
||||
- **COB Data**: 50 latest snapshots
|
||||
- **Model Predictions**: 20 recent predictions
|
||||
|
||||
## Usage
|
||||
|
||||
### **For Models**:
|
||||
```python
|
||||
# OLD: Direct data provider calls (inconsistent)
|
||||
data = data_provider.get_historical_data(symbol, timeframe, limit=300)
|
||||
|
||||
# NEW: Consistent data from orchestrator
|
||||
base_data = orchestrator.build_base_data_input(symbol)
|
||||
features = base_data.get_feature_vector() # Always 7850 features
|
||||
```
|
||||
|
||||
### **For Data Updates**:
|
||||
```python
|
||||
# Update FIFO queues with new data
|
||||
orchestrator.update_data_queue('ohlcv_1s', 'ETH/USDT', new_bar)
|
||||
orchestrator.update_data_queue('technical_indicators', 'ETH/USDT', indicators)
|
||||
```
|
||||
|
||||
### **For Monitoring**:
|
||||
```python
|
||||
# Check queue status
|
||||
status = orchestrator.get_queue_status()
|
||||
# {'ohlcv_1s': {'ETH/USDT': 450, 'BTC/USDT': 445}, ...}
|
||||
|
||||
# Verify minimum data
|
||||
has_data = orchestrator.ensure_minimum_data('ohlcv_1s', 'ETH/USDT', 100)
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
Run the test suite to verify the system:
|
||||
```bash
|
||||
python test_fifo_queues.py
|
||||
```
|
||||
|
||||
**Test Coverage**:
|
||||
- ✅ FIFO queue operations (add, retrieve, status)
|
||||
- ✅ Data queue filling with multiple timeframes
|
||||
- ✅ BaseDataInput building from queues
|
||||
- ✅ Consistent feature vector size (always 7850)
|
||||
- ✅ Thread safety under concurrent access
|
||||
- ✅ Minimum data requirement validation
|
||||
|
||||
## Monitoring
|
||||
|
||||
### **Queue Health**:
|
||||
```python
|
||||
status = orchestrator.get_queue_status()
|
||||
for data_type, symbols in status.items():
|
||||
for symbol, count in symbols.items():
|
||||
print(f"{data_type}/{symbol}: {count} items")
|
||||
```
|
||||
|
||||
### **Data Completeness**:
|
||||
```python
|
||||
# Check if ready for model inference
|
||||
ready = all([
|
||||
orchestrator.ensure_minimum_data('ohlcv_1s', 'ETH/USDT', 100),
|
||||
orchestrator.ensure_minimum_data('ohlcv_1m', 'ETH/USDT', 50),
|
||||
orchestrator.ensure_minimum_data('ohlcv_1h', 'ETH/USDT', 20),
|
||||
orchestrator.ensure_minimum_data('ohlcv_1d', 'ETH/USDT', 10)
|
||||
])
|
||||
```
|
||||
|
||||
### **Feature Vector Validation**:
|
||||
```python
|
||||
base_data = orchestrator.build_base_data_input('ETH/USDT')
|
||||
if base_data:
|
||||
features = base_data.get_feature_vector()
|
||||
assert len(features) == 7850, f"Feature size mismatch: {len(features)}"
|
||||
```
|
||||
|
||||
## Result
|
||||
|
||||
The FIFO queue system eliminates the network rebuilding issue by ensuring:
|
||||
|
||||
1. **Consistent input dimensions** - Always 7850 features
|
||||
2. **Complete data availability** - Minimum requirements guaranteed
|
||||
3. **Thread-safe operations** - No race conditions
|
||||
4. **Efficient data access** - In-memory queues vs. database calls
|
||||
5. **Stable model architecture** - No runtime network changes
|
||||
|
||||
**Before**:
|
||||
```
|
||||
2025-07-25 23:53:33,053 - INFO - Rebuilding network for new feature dimension: 300 (was 7850)
|
||||
2025-07-25 23:53:33,969 - INFO - Rebuilding network for new feature dimension: 7850 (was 300)
|
||||
```
|
||||
|
||||
**After**:
|
||||
```
|
||||
2025-07-25 23:53:33,053 - INFO - CNN prediction: BUY (conf=0.724) using 7850 features
|
||||
2025-07-25 23:53:34,012 - INFO - CNN prediction: HOLD (conf=0.651) using 7850 features
|
||||
```
|
||||
|
||||
The system now provides stable, consistent data flow that prevents the CNN from rebuilding its architecture at runtime.
|
||||
280
docs/logging_system_upgrade.md
Normal file
280
docs/logging_system_upgrade.md
Normal file
@@ -0,0 +1,280 @@
|
||||
# Trading System Logging Upgrade
|
||||
|
||||
## Overview
|
||||
|
||||
This upgrade implements a comprehensive logging and metadata management system that addresses the key issues:
|
||||
|
||||
1. **Eliminates scattered "No checkpoints found" logs** during runtime
|
||||
2. **Fast checkpoint metadata access** without loading full models
|
||||
3. **Centralized inference logging** with database and text file storage
|
||||
4. **Structured tracking** of model performance and checkpoints
|
||||
|
||||
## Key Components
|
||||
|
||||
### 1. Database Manager (`utils/database_manager.py`)
|
||||
|
||||
**Purpose**: SQLite-based storage for structured data
|
||||
|
||||
**Features**:
|
||||
- Inference records logging with deduplication
|
||||
- Checkpoint metadata storage (separate from model weights)
|
||||
- Model performance tracking
|
||||
- Fast queries without loading model files
|
||||
|
||||
**Tables**:
|
||||
- `inference_records`: All model predictions with metadata
|
||||
- `checkpoint_metadata`: Checkpoint info without model weights
|
||||
- `model_performance`: Daily aggregated statistics
|
||||
|
||||
### 2. Inference Logger (`utils/inference_logger.py`)
|
||||
|
||||
**Purpose**: Centralized logging for all model inferences
|
||||
|
||||
**Features**:
|
||||
- Single function call replaces scattered `logger.info()` calls
|
||||
- Automatic feature hashing for deduplication
|
||||
- Memory usage tracking
|
||||
- Processing time measurement
|
||||
- Dual storage (database + text files)
|
||||
|
||||
**Usage**:
|
||||
```python
|
||||
from utils.inference_logger import log_model_inference
|
||||
|
||||
log_model_inference(
|
||||
model_name="dqn_agent",
|
||||
symbol="ETH/USDT",
|
||||
action="BUY",
|
||||
confidence=0.85,
|
||||
probabilities={"BUY": 0.85, "SELL": 0.10, "HOLD": 0.05},
|
||||
input_features=features_array,
|
||||
processing_time_ms=12.5,
|
||||
checkpoint_id="dqn_agent_20250725_143500"
|
||||
)
|
||||
```
|
||||
|
||||
### 3. Text Logger (`utils/text_logger.py`)
|
||||
|
||||
**Purpose**: Human-readable log files for tracking
|
||||
|
||||
**Features**:
|
||||
- Separate files for different event types
|
||||
- Clean, tabular format
|
||||
- Automatic cleanup of old entries
|
||||
- Easy to read and grep
|
||||
|
||||
**Files**:
|
||||
- `logs/inference_records.txt`: All model predictions
|
||||
- `logs/checkpoint_events.txt`: Save/load events
|
||||
- `logs/system_events.txt`: General system events
|
||||
|
||||
### 4. Enhanced Checkpoint Manager (`utils/checkpoint_manager.py`)
|
||||
|
||||
**Purpose**: Improved checkpoint handling with metadata separation
|
||||
|
||||
**Features**:
|
||||
- Database-backed metadata storage
|
||||
- Fast metadata queries without loading models
|
||||
- Eliminates "No checkpoints found" spam
|
||||
- Backward compatibility with existing code
|
||||
|
||||
## Benefits
|
||||
|
||||
### 1. Performance Improvements
|
||||
|
||||
**Before**: Loading full checkpoint just to get metadata
|
||||
```python
|
||||
# Old way - loads entire model!
|
||||
checkpoint_path, metadata = load_best_checkpoint("dqn_agent")
|
||||
loss = metadata.loss # Expensive operation
|
||||
```
|
||||
|
||||
**After**: Fast metadata access from database
|
||||
```python
|
||||
# New way - database query only
|
||||
metadata = db_manager.get_best_checkpoint_metadata("dqn_agent")
|
||||
loss = metadata.performance_metrics['loss'] # Fast!
|
||||
```
|
||||
|
||||
### 2. Cleaner Runtime Logs
|
||||
|
||||
**Before**: Scattered logs everywhere
|
||||
```
|
||||
2025-07-25 14:34:39,749 - utils.checkpoint_manager - INFO - No checkpoints found for dqn_agent
|
||||
2025-07-25 14:34:39,754 - utils.checkpoint_manager - INFO - No checkpoints found for enhanced_cnn
|
||||
2025-07-25 14:34:39,756 - utils.checkpoint_manager - INFO - No checkpoints found for extrema_trainer
|
||||
```
|
||||
|
||||
**After**: Clean, structured logging
|
||||
```
|
||||
2025-07-25 14:34:39 | dqn_agent | ETH/USDT | BUY | conf=0.850 | time= 12.5ms [checkpoint: dqn_agent_20250725_143500]
|
||||
2025-07-25 14:34:40 | enhanced_cnn | ETH/USDT | HOLD | conf=0.720 | time= 8.2ms [checkpoint: enhanced_cnn_20250725_143501]
|
||||
```
|
||||
|
||||
### 3. Structured Data Storage
|
||||
|
||||
**Database Schema**:
|
||||
```sql
|
||||
-- Fast metadata queries
|
||||
SELECT * FROM checkpoint_metadata WHERE model_name = 'dqn_agent' AND is_active = TRUE;
|
||||
|
||||
-- Performance analysis
|
||||
SELECT model_name, AVG(confidence), COUNT(*)
|
||||
FROM inference_records
|
||||
WHERE timestamp > datetime('now', '-24 hours')
|
||||
GROUP BY model_name;
|
||||
```
|
||||
|
||||
### 4. Easy Integration
|
||||
|
||||
**In Model Code**:
|
||||
```python
|
||||
# Replace scattered logging
|
||||
# OLD: logger.info(f"DQN prediction: {action} confidence={conf}")
|
||||
|
||||
# NEW: Centralized logging
|
||||
self.orchestrator.log_model_inference(
|
||||
model_name="dqn_agent",
|
||||
symbol=symbol,
|
||||
action=action,
|
||||
confidence=confidence,
|
||||
probabilities=probs,
|
||||
input_features=features,
|
||||
processing_time_ms=processing_time
|
||||
)
|
||||
```
|
||||
|
||||
## Implementation Guide
|
||||
|
||||
### 1. Update Model Classes
|
||||
|
||||
Add inference logging to prediction methods:
|
||||
|
||||
```python
|
||||
class DQNAgent:
|
||||
def predict(self, state):
|
||||
start_time = time.time()
|
||||
|
||||
# Your prediction logic here
|
||||
action = self._predict_action(state)
|
||||
confidence = self._calculate_confidence()
|
||||
|
||||
processing_time = (time.time() - start_time) * 1000
|
||||
|
||||
# Log the inference
|
||||
self.orchestrator.log_model_inference(
|
||||
model_name="dqn_agent",
|
||||
symbol=self.symbol,
|
||||
action=action,
|
||||
confidence=confidence,
|
||||
probabilities=self.action_probabilities,
|
||||
input_features=state,
|
||||
processing_time_ms=processing_time,
|
||||
checkpoint_id=self.current_checkpoint_id
|
||||
)
|
||||
|
||||
return action
|
||||
```
|
||||
|
||||
### 2. Update Checkpoint Saving
|
||||
|
||||
Use the enhanced checkpoint manager:
|
||||
|
||||
```python
|
||||
from utils.checkpoint_manager import save_checkpoint
|
||||
|
||||
# Save with metadata
|
||||
checkpoint_metadata = save_checkpoint(
|
||||
model=self.model,
|
||||
model_name="dqn_agent",
|
||||
model_type="rl",
|
||||
performance_metrics={"loss": 0.0234, "accuracy": 0.87},
|
||||
training_metadata={"epochs": 100, "lr": 0.001}
|
||||
)
|
||||
```
|
||||
|
||||
### 3. Fast Metadata Access
|
||||
|
||||
Get checkpoint info without loading models:
|
||||
|
||||
```python
|
||||
# Fast metadata access
|
||||
metadata = orchestrator.get_checkpoint_metadata_fast("dqn_agent")
|
||||
if metadata:
|
||||
current_loss = metadata.performance_metrics['loss']
|
||||
checkpoint_id = metadata.checkpoint_id
|
||||
```
|
||||
|
||||
## Migration Steps
|
||||
|
||||
1. **Install new dependencies** (if any)
|
||||
2. **Update model classes** to use centralized logging
|
||||
3. **Replace checkpoint loading** with database queries where possible
|
||||
4. **Remove scattered logger.info()** calls for inferences
|
||||
5. **Test with demo script**: `python demo_logging_system.py`
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
utils/
|
||||
├── database_manager.py # SQLite database management
|
||||
├── inference_logger.py # Centralized inference logging
|
||||
├── text_logger.py # Human-readable text logs
|
||||
└── checkpoint_manager.py # Enhanced checkpoint handling
|
||||
|
||||
logs/ # Text log files
|
||||
├── inference_records.txt
|
||||
├── checkpoint_events.txt
|
||||
└── system_events.txt
|
||||
|
||||
data/
|
||||
└── trading_system.db # SQLite database
|
||||
|
||||
demo_logging_system.py # Demonstration script
|
||||
```
|
||||
|
||||
## Monitoring and Maintenance
|
||||
|
||||
### Daily Tasks
|
||||
- Check `logs/inference_records.txt` for recent activity
|
||||
- Monitor database size: `ls -lh data/trading_system.db`
|
||||
|
||||
### Weekly Tasks
|
||||
- Run cleanup: `inference_logger.cleanup_old_logs(days_to_keep=30)`
|
||||
- Check model performance trends in database
|
||||
|
||||
### Monthly Tasks
|
||||
- Archive old log files
|
||||
- Analyze model performance statistics
|
||||
- Review checkpoint storage usage
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **Database locked**: Multiple processes accessing SQLite
|
||||
- Solution: Use connection timeout and proper context managers
|
||||
|
||||
2. **Log files growing too large**:
|
||||
- Solution: Run `text_logger.cleanup_old_logs(max_lines=10000)`
|
||||
|
||||
3. **Missing checkpoint metadata**:
|
||||
- Solution: System falls back to file-based approach automatically
|
||||
|
||||
### Debug Commands
|
||||
|
||||
```python
|
||||
# Check database status
|
||||
db_manager = get_database_manager()
|
||||
checkpoints = db_manager.list_checkpoints("dqn_agent")
|
||||
|
||||
# Check recent inferences
|
||||
inference_logger = get_inference_logger()
|
||||
stats = inference_logger.get_model_stats("dqn_agent", hours=24)
|
||||
|
||||
# View text logs
|
||||
text_logger = get_text_logger()
|
||||
recent = text_logger.get_recent_inferences(lines=50)
|
||||
```
|
||||
|
||||
This upgrade provides a solid foundation for tracking model performance, eliminating log spam, and enabling fast metadata access without the overhead of loading full model checkpoints.
|
||||
@@ -16,6 +16,18 @@ ETH:
|
||||
300s of 1s OHLCV data (5 min)
|
||||
300 OHLCV + indicatros bars of each 1m 1h 1d and 1s BTC
|
||||
|
||||
so:
|
||||
|
||||
# Standardized input for all models:
|
||||
{
|
||||
'primary_symbol': 'ETH/USDT',
|
||||
'reference_symbol': 'BTC/USDT',
|
||||
'eth_data': {'ETH_1s': df, 'ETH_1m': df, 'ETH_1h': df, 'ETH_1d': df},
|
||||
'btc_data': {'BTC_1s': df},
|
||||
'current_prices': {'ETH': price, 'BTC': price},
|
||||
'data_completeness': {...}
|
||||
}
|
||||
|
||||
RL model should have also access of the last hidden layers of the CNN model where patterns are learned. it can be empty if CNN model is not active or missing. as well as the output (predictions) of the CNN model for each timeframe (1s 1m 1h 1d) and next expected pivot point
|
||||
|
||||
## CNN model
|
||||
|
||||
Reference in New Issue
Block a user