2 Commits

Author SHA1 Message Date
6b9482d2be pivots 2025-07-17 02:15:24 +03:00
b4e592b406 kiro tasks 2025-07-17 01:02:16 +03:00
6 changed files with 1119 additions and 21 deletions

View File

@ -12,6 +12,7 @@ The Multi-Modal Trading System is an advanced algorithmic trading platform that
#### Acceptance Criteria
0. NEVER USE GENERATED/SYNTHETIC DATA or mock implementations and UI. If somethings is not implemented yet, it should be obvious.
1. WHEN the system starts THEN it SHALL collect and process data for both ETH and BTC symbols.
2. WHEN collecting data THEN the system SHALL store the following for the primary symbol (ETH):
- 300 seconds of raw tick data - price and COB snapshot for all prices +- 1% on fine reslolution buckets (1$ for ETH, 10$ for BTC)
@ -24,7 +25,7 @@ The Multi-Modal Trading System is an advanced algorithmic trading platform that
7. IF tick data is not available THEN the system SHALL substitute with the lowest available timeframe data.
8. WHEN normalizing data THEN the system SHALL normalize to the max and min of the highest timeframe to maintain relationships between different timeframes.
9. data is cached for longer (let's start with double the model inputs so 600 bars) to support performing backtesting when we know the current predictions outcomes so we can generate test cases.
10. In general all models have access to the whole data we collect in a central data provider implementation. only some are specialized.
10. In general all models have access to the whole data we collect in a central data provider implementation. only some are specialized. All models should also take as input the last output of evey other model (also cached in the data provider). there should be a room for adding more models in the other models data input so we can extend the system without having to loose existing models and trained W&B
### Requirement 2: CNN Model Implementation

View File

@ -0,0 +1,247 @@
# Implementation Plan
## Data Provider and Processing
- [ ] 1. Enhance the existing DataProvider class
- Extend the current implementation in core/data_provider.py
- Ensure it supports all required timeframes (1s, 1m, 1h, 1d)
- Implement better error handling and fallback mechanisms
- _Requirements: 1.1, 1.2, 1.3, 1.6_
- [ ] 1.1. Implement Williams Market Structure pivot point calculation
- Create a dedicated method for identifying pivot points
- Implement the recursive pivot point calculation as described
- Add unit tests to verify pivot point detection accuracy
- _Requirements: 1.5, 2.7_
- [ ] 1.2. Optimize data caching for better performance
- Implement efficient caching strategies for different timeframes
- Add cache invalidation mechanisms
- Ensure thread safety for cache access
- _Requirements: 1.6, 8.1_
- [ ] 1.3. Enhance real-time data streaming
- Improve WebSocket connection management
- Implement reconnection strategies
- Add data validation to ensure data integrity
- _Requirements: 1.6, 8.5_
- [ ] 1.4. Implement data normalization
- Normalize data based on the highest timeframe
- Ensure relationships between different timeframes are maintained
- Add unit tests to verify normalization correctness
- _Requirements: 1.8, 2.1_
## CNN Model Implementation
- [ ] 2. Design and implement the CNN model architecture
- Create a CNNModel class that accepts multi-timeframe and multi-symbol data
- Implement the model using PyTorch or TensorFlow
- Design the architecture with convolutional, LSTM/GRU, and attention layers
- _Requirements: 2.1, 2.2, 2.8_
- [ ] 2.1. Implement pivot point prediction
- Create a PivotPointPredictor class
- Implement methods to predict pivot points for each timeframe
- Add confidence score calculation for predictions
- _Requirements: 2.2, 2.3, 2.6_
- [ ] 2.2. Implement CNN training pipeline
- Create a CNNTrainer class
- Implement methods for training the model on historical data
- Add mechanisms to trigger training when new pivot points are detected
- _Requirements: 2.4, 2.5, 5.2, 5.3_
- [ ] 2.3. Implement CNN inference pipeline
- Create methods for real-time inference
- Ensure hidden layer states are accessible for the RL model
- Optimize for performance to minimize latency
- _Requirements: 2.2, 2.6, 2.8_
- [ ] 2.4. Implement model evaluation and validation
- Create methods to evaluate model performance
- Implement metrics for prediction accuracy
- Add validation against historical pivot points
- _Requirements: 2.5, 5.8_
## RL Model Implementation
- [ ] 3. Design and implement the RL model architecture
- Create an RLModel class that accepts market data and CNN outputs
- Implement the model using PyTorch or TensorFlow
- Design the architecture with state representation, action space, and reward function
- _Requirements: 3.1, 3.2, 3.7_
- [ ] 3.1. Implement trading action generation
- Create a TradingActionGenerator class
- Implement methods to generate buy/sell recommendations
- Add confidence score calculation for actions
- _Requirements: 3.2, 3.7_
- [ ] 3.2. Implement RL training pipeline
- Create an RLTrainer class
- Implement methods for training the model on historical data
- Add experience replay for improved sample efficiency
- _Requirements: 3.3, 3.5, 5.4_
- [ ] 3.3. Implement RL inference pipeline
- Create methods for real-time inference
- Optimize for performance to minimize latency
- Ensure proper handling of CNN inputs
- _Requirements: 3.1, 3.2, 3.4_
- [ ] 3.4. Implement model evaluation and validation
- Create methods to evaluate model performance
- Implement metrics for trading performance
- Add validation against historical trading opportunities
- _Requirements: 3.3, 5.8_
## Orchestrator Implementation
- [ ] 4. Design and implement the orchestrator architecture
- Create an Orchestrator class that accepts inputs from CNN and RL models
- Implement the Mixture of Experts (MoE) approach
- Design the architecture with gating network and decision network
- _Requirements: 4.1, 4.2, 4.5_
- [ ] 4.1. Implement decision-making logic
- Create a DecisionMaker class
- Implement methods to make final trading decisions
- Add confidence-based filtering
- _Requirements: 4.2, 4.3, 4.4_
- [ ] 4.2. Implement MoE gateway
- Create a MoEGateway class
- Implement methods to determine which expert to trust
- Add mechanisms for future model integration
- _Requirements: 4.5, 8.2_
- [ ] 4.3. Implement configurable thresholds
- Add parameters for entering and exiting positions
- Implement methods to adjust thresholds dynamically
- Add validation to ensure thresholds are within reasonable ranges
- _Requirements: 4.8, 6.7_
- [ ] 4.4. Implement model evaluation and validation
- Create methods to evaluate orchestrator performance
- Implement metrics for decision quality
- Add validation against historical trading decisions
- _Requirements: 4.6, 5.8_
## Trading Executor Implementation
- [ ] 5. Design and implement the trading executor
- Create a TradingExecutor class that accepts trading actions from the orchestrator
- Implement order execution through brokerage APIs
- Add order lifecycle management
- _Requirements: 7.1, 7.2, 8.6_
- [ ] 5.1. Implement brokerage API integrations
- Create a BrokerageAPI interface
- Implement concrete classes for MEXC and Binance
- Add error handling and retry mechanisms
- _Requirements: 7.1, 7.2, 8.6_
- [ ] 5.2. Implement order management
- Create an OrderManager class
- Implement methods for creating, updating, and canceling orders
- Add order tracking and status updates
- _Requirements: 7.1, 7.2, 8.6_
- [ ] 5.3. Implement error handling
- Add comprehensive error handling for API failures
- Implement circuit breakers for extreme market conditions
- Add logging and notification mechanisms
- _Requirements: 7.1, 7.2, 8.6_
## Risk Manager Implementation
- [ ] 6. Design and implement the risk manager
- Create a RiskManager class
- Implement risk parameter management
- Add risk metric calculation
- _Requirements: 7.1, 7.3, 7.4_
- [ ] 6.1. Implement stop-loss functionality
- Create a StopLossManager class
- Implement methods for creating and managing stop-loss orders
- Add mechanisms to automatically close positions when stop-loss is triggered
- _Requirements: 7.1, 7.2_
- [ ] 6.2. Implement position sizing
- Create a PositionSizer class
- Implement methods for calculating position sizes based on risk parameters
- Add validation to ensure position sizes are within limits
- _Requirements: 7.3, 7.7_
- [ ] 6.3. Implement risk metrics
- Add methods to calculate risk metrics (drawdown, VaR, etc.)
- Implement real-time risk monitoring
- Add alerts for high-risk situations
- _Requirements: 7.4, 7.5, 7.6, 7.8_
## Dashboard Implementation
- [ ] 7. Design and implement the dashboard UI
- Create a Dashboard class
- Implement the web-based UI using Flask/Dash
- Add real-time updates using WebSockets
- _Requirements: 6.1, 6.8_
- [ ] 7.1. Implement chart management
- Create a ChartManager class
- Implement methods for creating and updating charts
- Add interactive features (zoom, pan, etc.)
- _Requirements: 6.1, 6.2_
- [ ] 7.2. Implement control panel
- Create a ControlPanel class
- Implement start/stop toggles for system processes
- Add sliders for adjusting buy/sell thresholds
- _Requirements: 6.6, 6.7_
- [ ] 7.3. Implement system status display
- Add methods to display training progress
- Implement model performance metrics visualization
- Add real-time system status updates
- _Requirements: 6.5, 5.6_
- [ ] 7.4. Implement server-side processing
- Ensure all processes run on the server without requiring the dashboard to be open
- Implement background tasks for model training and inference
- Add mechanisms to persist system state
- _Requirements: 6.8, 5.5_
## Integration and Testing
- [ ] 8. Integrate all components
- Connect the data provider to the CNN and RL models
- Connect the CNN and RL models to the orchestrator
- Connect the orchestrator to the trading executor
- _Requirements: 8.1, 8.2, 8.3_
- [ ] 8.1. Implement comprehensive unit tests
- Create unit tests for each component
- Implement test fixtures and mocks
- Add test coverage reporting
- _Requirements: 8.1, 8.2, 8.3_
- [ ] 8.2. Implement integration tests
- Create tests for component interactions
- Implement end-to-end tests
- Add performance benchmarks
- _Requirements: 8.1, 8.2, 8.3_
- [ ] 8.3. Implement backtesting framework
- Create a backtesting environment
- Implement methods to replay historical data
- Add performance metrics calculation
- _Requirements: 5.8, 8.1_
- [ ] 8.4. Optimize performance
- Profile the system to identify bottlenecks
- Implement optimizations for critical paths
- Add caching and parallelization where appropriate
- _Requirements: 8.1, 8.2, 8.3_

View File

@ -34,6 +34,7 @@ from collections import deque
from .config import get_config
from .tick_aggregator import RealTimeTickAggregator, RawTick, OHLCVBar
from .cnn_monitor import log_cnn_prediction
from .williams_market_structure import WilliamsMarketStructure, PivotPoint, TrendLevel
logger = logging.getLogger(__name__)
@ -182,6 +183,16 @@ class DataProvider:
'1h': 3600, '4h': 14400, '1d': 86400
}
# Williams Market Structure integration
self.williams_structure: Dict[str, WilliamsMarketStructure] = {}
for symbol in self.symbols:
self.williams_structure[symbol] = WilliamsMarketStructure(min_pivot_distance=3)
# Pivot point caching
self.pivot_points_cache: Dict[str, Dict[int, TrendLevel]] = {} # {symbol: {level: TrendLevel}}
self.last_pivot_calculation: Dict[str, datetime] = {}
self.pivot_calculation_interval = timedelta(minutes=5) # Recalculate every 5 minutes
# Load existing pivot bounds from cache
self._load_all_pivot_bounds()
@ -189,6 +200,7 @@ class DataProvider:
logger.info(f"Timeframes: {self.timeframes}")
logger.info("Centralized data distribution enabled")
logger.info("Pivot-based normalization system enabled")
logger.info("Williams Market Structure integration enabled")
# Rate limiting
self.last_request_time = {}
@ -1613,6 +1625,151 @@ class DataProvider:
logger.error(f"Error getting current price for {symbol}: {e}")
return None
def calculate_williams_pivot_points(self, symbol: str, force_recalculate: bool = False) -> Dict[int, TrendLevel]:
"""
Calculate Williams Market Structure pivot points for a symbol
Args:
symbol: Trading symbol (e.g., 'ETH/USDT')
force_recalculate: Force recalculation even if cache is fresh
Returns:
Dictionary of trend levels with pivot points
"""
try:
# Check if we need to recalculate
now = datetime.now()
if (not force_recalculate and
symbol in self.last_pivot_calculation and
now - self.last_pivot_calculation[symbol] < self.pivot_calculation_interval):
# Return cached results
return self.pivot_points_cache.get(symbol, {})
# Get 1s OHLCV data for Williams Market Structure calculation
df_1s = self.get_historical_data(symbol, '1s', limit=1000)
if df_1s is None or len(df_1s) < 50:
logger.warning(f"Insufficient 1s data for Williams pivot calculation: {symbol}")
return {}
# Convert DataFrame to numpy array for Williams calculation
# Format: [timestamp_ms, open, high, low, close, volume]
ohlcv_array = np.column_stack([
df_1s.index.astype(np.int64) // 10**6, # Convert to milliseconds
df_1s['open'].values,
df_1s['high'].values,
df_1s['low'].values,
df_1s['close'].values,
df_1s['volume'].values
])
# Calculate recursive pivot points using Williams Market Structure
williams = self.williams_structure[symbol]
pivot_levels = williams.calculate_recursive_pivot_points(ohlcv_array)
# Cache the results
self.pivot_points_cache[symbol] = pivot_levels
self.last_pivot_calculation[symbol] = now
logger.debug(f"Calculated Williams pivot points for {symbol}: {len(pivot_levels)} levels")
return pivot_levels
except Exception as e:
logger.error(f"Error calculating Williams pivot points for {symbol}: {e}")
return {}
def get_pivot_features_for_ml(self, symbol: str) -> np.ndarray:
"""
Get pivot point features for machine learning models
Returns a 250-element feature vector containing:
- Recent pivot points (price, strength, type) for each level
- Trend direction and strength for each level
- Time since last pivot for each level
"""
try:
# Ensure we have fresh pivot points
pivot_levels = self.calculate_williams_pivot_points(symbol)
if not pivot_levels:
logger.warning(f"No pivot points available for {symbol}")
return np.zeros(250, dtype=np.float32)
# Use Williams Market Structure to extract ML features
williams = self.williams_structure[symbol]
features = williams.get_pivot_features_for_ml(symbol)
return features
except Exception as e:
logger.error(f"Error getting pivot features for ML: {e}")
return np.zeros(250, dtype=np.float32)
def get_market_structure_summary(self, symbol: str) -> Dict[str, Any]:
"""
Get current market structure summary for dashboard display
Returns:
Dictionary containing market structure information
"""
try:
# Ensure we have fresh pivot points
pivot_levels = self.calculate_williams_pivot_points(symbol)
if not pivot_levels:
return {
'symbol': symbol,
'levels': {},
'overall_trend': 'sideways',
'overall_strength': 0.0,
'last_update': datetime.now().isoformat(),
'error': 'No pivot points available'
}
# Use Williams Market Structure to get summary
williams = self.williams_structure[symbol]
structure = williams.get_current_market_structure()
structure['symbol'] = symbol
return structure
except Exception as e:
logger.error(f"Error getting market structure summary for {symbol}: {e}")
return {
'symbol': symbol,
'levels': {},
'overall_trend': 'sideways',
'overall_strength': 0.0,
'last_update': datetime.now().isoformat(),
'error': str(e)
}
def get_recent_pivot_points(self, symbol: str, level: int = 1, count: int = 10) -> List[PivotPoint]:
"""
Get recent pivot points for a specific level
Args:
symbol: Trading symbol
level: Pivot level (1-5)
count: Number of recent pivots to return
Returns:
List of recent pivot points
"""
try:
pivot_levels = self.calculate_williams_pivot_points(symbol)
if level not in pivot_levels:
return []
trend_level = pivot_levels[level]
recent_pivots = trend_level.pivot_points[-count:] if len(trend_level.pivot_points) >= count else trend_level.pivot_points
return recent_pivots
except Exception as e:
logger.error(f"Error getting recent pivot points for {symbol} level {level}: {e}")
return []
def get_price_at_index(self, symbol: str, index: int, timeframe: str = '1m') -> Optional[float]:
"""Get price at specific index for backtesting"""
try:

View File

@ -136,6 +136,11 @@ class TradingOrchestrator:
self.recent_decisions: Dict[str, List[TradingDecision]] = {} # {symbol: List[TradingDecision]}
self.model_performance: Dict[str, Dict[str, Any]] = {} # {model_name: {'correct': int, 'total': int, 'accuracy': float}}
# Signal rate limiting to prevent spam
self.last_signal_time: Dict[str, Dict[str, datetime]] = {} # {symbol: {action: datetime}}
self.min_signal_interval = timedelta(seconds=30) # Minimum 30 seconds between same signals
self.last_confirmed_signal: Dict[str, Dict[str, Any]] = {} # {symbol: {action, timestamp, confidence}}
# Signal accumulation for trend confirmation
self.signal_accumulator: Dict[str, List[Dict]] = {} # {symbol: List[signal_data]}
self.required_confirmations = 3 # Number of consistent signals needed
@ -871,6 +876,22 @@ class TradingOrchestrator:
'CNN': self.config.orchestrator.get('cnn_weight', 0.7),
'RL': self.config.orchestrator.get('rl_weight', 0.3)
}
# Add weights for specific models if they exist
if hasattr(self, 'cnn_model') and self.cnn_model:
self.model_weights["enhanced_cnn"] = 0.4
# Only add DQN agent weight if it exists
if hasattr(self, 'rl_agent') and self.rl_agent:
self.model_weights["dqn_agent"] = 0.3
# Add COB RL model weight if it exists
if hasattr(self, 'cob_rl_agent') and self.cob_rl_agent:
self.model_weights["cob_rl_model"] = 0.2
# Add extrema trainer weight if it exists
if hasattr(self, 'extrema_trainer') and self.extrema_trainer:
self.model_weights["extrema_trainer"] = 0.15
def register_model(self, model: ModelInterface, weight: Optional[float] = None) -> bool:
"""Register a new model with the orchestrator"""
@ -1960,10 +1981,27 @@ class TradingOrchestrator:
logger.info("Trading executor set for position tracking and P&L feedback")
def _check_signal_confirmation(self, symbol: str, signal_data: Dict) -> Optional[str]:
"""Check if we have enough signal confirmations for trend confirmation"""
"""Check if we have enough signal confirmations for trend confirmation with rate limiting"""
try:
# Clean up expired signals
current_time = signal_data['timestamp']
action = signal_data['action']
# Initialize signal tracking for this symbol if needed
if symbol not in self.last_signal_time:
self.last_signal_time[symbol] = {}
if symbol not in self.last_confirmed_signal:
self.last_confirmed_signal[symbol] = {}
# RATE LIMITING: Check if we recently confirmed the same signal
if action in self.last_confirmed_signal[symbol]:
last_confirmed = self.last_confirmed_signal[symbol][action]
time_since_last = current_time - last_confirmed['timestamp']
if time_since_last < self.min_signal_interval:
logger.debug(f"Rate limiting: {action} signal for {symbol} too recent "
f"({time_since_last.total_seconds():.1f}s < {self.min_signal_interval.total_seconds()}s)")
return None
# Clean up expired signals
self.signal_accumulator[symbol] = [
s for s in self.signal_accumulator[symbol]
if (current_time - s['timestamp']).total_seconds() < self.signal_timeout_seconds
@ -1982,8 +2020,8 @@ class TradingOrchestrator:
# Count action consensus
action_counts = {}
for action in actions:
action_counts[action] = action_counts.get(action, 0) + 1
for action_item in actions:
action_counts[action_item] = action_counts.get(action_item, 0) + 1
# Find dominant action
dominant_action = max(action_counts, key=action_counts.get)
@ -1991,8 +2029,24 @@ class TradingOrchestrator:
# Require at least 2/3 consensus
if consensus_count >= max(2, self.required_confirmations * 0.67):
# ADDITIONAL RATE LIMITING: Don't confirm if we just confirmed the same action
if dominant_action in self.last_confirmed_signal[symbol]:
last_confirmed = self.last_confirmed_signal[symbol][dominant_action]
time_since_last = current_time - last_confirmed['timestamp']
if time_since_last < self.min_signal_interval:
logger.debug(f"Rate limiting: Preventing duplicate {dominant_action} confirmation for {symbol}")
return None
# Record this confirmation
self.last_confirmed_signal[symbol][dominant_action] = {
'timestamp': current_time,
'confidence': signal_data['confidence']
}
# Clear accumulator after confirmation
self.signal_accumulator[symbol] = []
logger.info(f"Signal confirmed after rate limiting: {dominant_action} for {symbol}")
return dominant_action
return None

View File

@ -0,0 +1,555 @@
"""
Williams Market Structure Implementation
This module implements Larry Williams' market structure analysis with recursive pivot points.
The system identifies swing highs and swing lows, then uses these pivot points to determine
higher-level trends recursively.
Key Features:
- Recursive pivot point calculation (5 levels)
- Swing high/low identification
- Trend direction and strength analysis
- Integration with CNN model for pivot prediction
"""
import logging
import numpy as np
import pandas as pd
from datetime import datetime, timedelta
from typing import Dict, List, Optional, Tuple, Any
from dataclasses import dataclass, field
from collections import deque
logger = logging.getLogger(__name__)
@dataclass
class PivotPoint:
"""Represents a pivot point in the market structure"""
timestamp: datetime
price: float
pivot_type: str # 'high' or 'low'
level: int # Pivot level (1-5)
index: int # Index in the original data
strength: float = 0.0 # Strength of the pivot (0.0 to 1.0)
confirmed: bool = False # Whether the pivot is confirmed
@dataclass
class TrendLevel:
"""Represents a trend level in the Williams Market Structure"""
level: int
pivot_points: List[PivotPoint]
trend_direction: str # 'up', 'down', 'sideways'
trend_strength: float # 0.0 to 1.0
last_pivot_high: Optional[PivotPoint] = None
last_pivot_low: Optional[PivotPoint] = None
class WilliamsMarketStructure:
"""
Implementation of Larry Williams Market Structure Analysis
This class implements the recursive pivot point calculation system where:
1. Level 1: Direct swing highs/lows from 1s OHLCV data
2. Level 2-5: Recursive analysis using previous level's pivot points as "candles"
"""
def __init__(self, min_pivot_distance: int = 3):
"""
Initialize Williams Market Structure analyzer
Args:
min_pivot_distance: Minimum distance between pivot points
"""
self.min_pivot_distance = min_pivot_distance
self.pivot_levels: Dict[int, TrendLevel] = {}
self.max_levels = 5
logger.info(f"Williams Market Structure initialized with {self.max_levels} levels")
def calculate_recursive_pivot_points(self, ohlcv_data: np.ndarray) -> Dict[int, TrendLevel]:
"""
Calculate recursive pivot points following Williams Market Structure methodology
Args:
ohlcv_data: OHLCV data array with shape (N, 6) [timestamp, O, H, L, C, V]
Returns:
Dictionary of trend levels with pivot points
"""
try:
if len(ohlcv_data) < self.min_pivot_distance * 2 + 1:
logger.warning(f"Insufficient data for pivot calculation: {len(ohlcv_data)} bars")
return {}
# Convert to DataFrame for easier processing
df = pd.DataFrame(ohlcv_data, columns=['timestamp', 'open', 'high', 'low', 'close', 'volume'])
df['timestamp'] = pd.to_datetime(df['timestamp'], unit='ms')
# Initialize pivot levels
self.pivot_levels = {}
# Level 1: Calculate pivot points from raw OHLCV data
level_1_pivots = self._calculate_level_1_pivots(df)
if level_1_pivots:
self.pivot_levels[1] = TrendLevel(
level=1,
pivot_points=level_1_pivots,
trend_direction=self._determine_trend_direction(level_1_pivots),
trend_strength=self._calculate_trend_strength(level_1_pivots)
)
# Levels 2-5: Recursive calculation using previous level's pivots
for level in range(2, self.max_levels + 1):
higher_level_pivots = self._calculate_higher_level_pivots(level)
if higher_level_pivots:
self.pivot_levels[level] = TrendLevel(
level=level,
pivot_points=higher_level_pivots,
trend_direction=self._determine_trend_direction(higher_level_pivots),
trend_strength=self._calculate_trend_strength(higher_level_pivots)
)
else:
break # No more higher level pivots possible
logger.debug(f"Calculated {len(self.pivot_levels)} pivot levels")
return self.pivot_levels
except Exception as e:
logger.error(f"Error calculating recursive pivot points: {e}")
return {}
def _calculate_level_1_pivots(self, df: pd.DataFrame) -> List[PivotPoint]:
"""
Calculate Level 1 pivot points from raw OHLCV data
A swing high is a candle with lower highs on both sides
A swing low is a candle with higher lows on both sides
"""
pivots = []
try:
for i in range(self.min_pivot_distance, len(df) - self.min_pivot_distance):
current_high = df.iloc[i]['high']
current_low = df.iloc[i]['low']
current_timestamp = df.iloc[i]['timestamp']
# Check for swing high
is_swing_high = True
for j in range(i - self.min_pivot_distance, i + self.min_pivot_distance + 1):
if j != i and df.iloc[j]['high'] >= current_high:
is_swing_high = False
break
if is_swing_high:
pivot = PivotPoint(
timestamp=current_timestamp,
price=current_high,
pivot_type='high',
level=1,
index=i,
strength=self._calculate_pivot_strength(df, i, 'high'),
confirmed=True
)
pivots.append(pivot)
continue
# Check for swing low
is_swing_low = True
for j in range(i - self.min_pivot_distance, i + self.min_pivot_distance + 1):
if j != i and df.iloc[j]['low'] <= current_low:
is_swing_low = False
break
if is_swing_low:
pivot = PivotPoint(
timestamp=current_timestamp,
price=current_low,
pivot_type='low',
level=1,
index=i,
strength=self._calculate_pivot_strength(df, i, 'low'),
confirmed=True
)
pivots.append(pivot)
logger.debug(f"Level 1: Found {len(pivots)} pivot points")
return pivots
except Exception as e:
logger.error(f"Error calculating Level 1 pivots: {e}")
return []
def _calculate_higher_level_pivots(self, level: int) -> List[PivotPoint]:
"""
Calculate higher level pivot points using previous level's pivots as "candles"
This is the recursive part of Williams Market Structure where we treat
pivot points from the previous level as if they were OHLCV candles
"""
if level - 1 not in self.pivot_levels:
return []
previous_level_pivots = self.pivot_levels[level - 1].pivot_points
if len(previous_level_pivots) < self.min_pivot_distance * 2 + 1:
return []
pivots = []
try:
# Group pivots by type to find swing points
highs = [p for p in previous_level_pivots if p.pivot_type == 'high']
lows = [p for p in previous_level_pivots if p.pivot_type == 'low']
# Find swing highs among the high pivots
for i in range(self.min_pivot_distance, len(highs) - self.min_pivot_distance):
current_pivot = highs[i]
# Check if this high is surrounded by lower highs
is_swing_high = True
for j in range(i - self.min_pivot_distance, i + self.min_pivot_distance + 1):
if j != i and j < len(highs) and highs[j].price >= current_pivot.price:
is_swing_high = False
break
if is_swing_high:
pivot = PivotPoint(
timestamp=current_pivot.timestamp,
price=current_pivot.price,
pivot_type='high',
level=level,
index=current_pivot.index,
strength=current_pivot.strength * 0.8, # Reduce strength at higher levels
confirmed=True
)
pivots.append(pivot)
# Find swing lows among the low pivots
for i in range(self.min_pivot_distance, len(lows) - self.min_pivot_distance):
current_pivot = lows[i]
# Check if this low is surrounded by higher lows
is_swing_low = True
for j in range(i - self.min_pivot_distance, i + self.min_pivot_distance + 1):
if j != i and j < len(lows) and lows[j].price <= current_pivot.price:
is_swing_low = False
break
if is_swing_low:
pivot = PivotPoint(
timestamp=current_pivot.timestamp,
price=current_pivot.price,
pivot_type='low',
level=level,
index=current_pivot.index,
strength=current_pivot.strength * 0.8, # Reduce strength at higher levels
confirmed=True
)
pivots.append(pivot)
# Sort pivots by timestamp
pivots.sort(key=lambda x: x.timestamp)
logger.debug(f"Level {level}: Found {len(pivots)} pivot points")
return pivots
except Exception as e:
logger.error(f"Error calculating Level {level} pivots: {e}")
return []
def _calculate_pivot_strength(self, df: pd.DataFrame, index: int, pivot_type: str) -> float:
"""
Calculate the strength of a pivot point based on surrounding price action
Strength is determined by:
- Distance from surrounding highs/lows
- Volume at the pivot point
- Duration of the pivot formation
"""
try:
if pivot_type == 'high':
current_price = df.iloc[index]['high']
# Calculate average of surrounding highs
surrounding_prices = []
for i in range(max(0, index - self.min_pivot_distance),
min(len(df), index + self.min_pivot_distance + 1)):
if i != index:
surrounding_prices.append(df.iloc[i]['high'])
if surrounding_prices:
avg_surrounding = np.mean(surrounding_prices)
strength = min(1.0, (current_price - avg_surrounding) / avg_surrounding * 10)
else:
strength = 0.5
else: # pivot_type == 'low'
current_price = df.iloc[index]['low']
# Calculate average of surrounding lows
surrounding_prices = []
for i in range(max(0, index - self.min_pivot_distance),
min(len(df), index + self.min_pivot_distance + 1)):
if i != index:
surrounding_prices.append(df.iloc[i]['low'])
if surrounding_prices:
avg_surrounding = np.mean(surrounding_prices)
strength = min(1.0, (avg_surrounding - current_price) / avg_surrounding * 10)
else:
strength = 0.5
# Factor in volume if available
if 'volume' in df.columns and df.iloc[index]['volume'] > 0:
avg_volume = df['volume'].rolling(window=20, center=True).mean().iloc[index]
if avg_volume > 0:
volume_factor = min(2.0, df.iloc[index]['volume'] / avg_volume)
strength *= volume_factor
return max(0.0, min(1.0, strength))
except Exception as e:
logger.error(f"Error calculating pivot strength: {e}")
return 0.5
def _determine_trend_direction(self, pivots: List[PivotPoint]) -> str:
"""
Determine the overall trend direction based on pivot points
Trend is determined by comparing recent highs and lows:
- Uptrend: Higher highs and higher lows
- Downtrend: Lower highs and lower lows
- Sideways: Mixed or insufficient data
"""
if len(pivots) < 4:
return 'sideways'
try:
# Get recent pivots (last 10 or all if less than 10)
recent_pivots = pivots[-10:] if len(pivots) >= 10 else pivots
highs = [p for p in recent_pivots if p.pivot_type == 'high']
lows = [p for p in recent_pivots if p.pivot_type == 'low']
if len(highs) < 2 or len(lows) < 2:
return 'sideways'
# Sort by timestamp
highs.sort(key=lambda x: x.timestamp)
lows.sort(key=lambda x: x.timestamp)
# Check for higher highs and higher lows (uptrend)
higher_highs = highs[-1].price > highs[-2].price if len(highs) >= 2 else False
higher_lows = lows[-1].price > lows[-2].price if len(lows) >= 2 else False
# Check for lower highs and lower lows (downtrend)
lower_highs = highs[-1].price < highs[-2].price if len(highs) >= 2 else False
lower_lows = lows[-1].price < lows[-2].price if len(lows) >= 2 else False
if higher_highs and higher_lows:
return 'up'
elif lower_highs and lower_lows:
return 'down'
else:
return 'sideways'
except Exception as e:
logger.error(f"Error determining trend direction: {e}")
return 'sideways'
def _calculate_trend_strength(self, pivots: List[PivotPoint]) -> float:
"""
Calculate the strength of the current trend
Strength is based on:
- Consistency of pivot point progression
- Average strength of individual pivots
- Number of confirming pivots
"""
if not pivots:
return 0.0
try:
# Average individual pivot strengths
avg_pivot_strength = np.mean([p.strength for p in pivots])
# Factor in number of pivots (more pivots = stronger trend)
pivot_count_factor = min(1.0, len(pivots) / 10.0)
# Calculate consistency (how well pivots follow the trend)
trend_direction = self._determine_trend_direction(pivots)
consistency_score = self._calculate_trend_consistency(pivots, trend_direction)
# Combine factors
trend_strength = (avg_pivot_strength * 0.4 +
pivot_count_factor * 0.3 +
consistency_score * 0.3)
return max(0.0, min(1.0, trend_strength))
except Exception as e:
logger.error(f"Error calculating trend strength: {e}")
return 0.0
def _calculate_trend_consistency(self, pivots: List[PivotPoint], trend_direction: str) -> float:
"""
Calculate how consistently the pivots follow the expected trend direction
"""
if len(pivots) < 4 or trend_direction == 'sideways':
return 0.5
try:
highs = [p for p in pivots if p.pivot_type == 'high']
lows = [p for p in pivots if p.pivot_type == 'low']
if len(highs) < 2 or len(lows) < 2:
return 0.5
# Sort by timestamp
highs.sort(key=lambda x: x.timestamp)
lows.sort(key=lambda x: x.timestamp)
consistent_moves = 0
total_moves = 0
# Check high-to-high moves
for i in range(1, len(highs)):
total_moves += 1
if trend_direction == 'up' and highs[i].price > highs[i-1].price:
consistent_moves += 1
elif trend_direction == 'down' and highs[i].price < highs[i-1].price:
consistent_moves += 1
# Check low-to-low moves
for i in range(1, len(lows)):
total_moves += 1
if trend_direction == 'up' and lows[i].price > lows[i-1].price:
consistent_moves += 1
elif trend_direction == 'down' and lows[i].price < lows[i-1].price:
consistent_moves += 1
if total_moves == 0:
return 0.5
return consistent_moves / total_moves
except Exception as e:
logger.error(f"Error calculating trend consistency: {e}")
return 0.5
def get_pivot_features_for_ml(self, symbol: str = "ETH/USDT") -> np.ndarray:
"""
Extract pivot point features for machine learning models
Returns a feature vector containing:
- Recent pivot points (price, strength, type)
- Trend direction and strength for each level
- Time since last pivot for each level
Total features: 250 (50 features per level * 5 levels)
"""
features = []
try:
for level in range(1, self.max_levels + 1):
level_features = []
if level in self.pivot_levels:
trend_level = self.pivot_levels[level]
pivots = trend_level.pivot_points
# Get last 5 pivots for this level
recent_pivots = pivots[-5:] if len(pivots) >= 5 else pivots
# Pad with zeros if we have fewer than 5 pivots
while len(recent_pivots) < 5:
recent_pivots.insert(0, PivotPoint(
timestamp=datetime.now(),
price=0.0,
pivot_type='high',
level=level,
index=0,
strength=0.0
))
# Extract features for each pivot (8 features per pivot)
for pivot in recent_pivots:
level_features.extend([
pivot.price,
pivot.strength,
1.0 if pivot.pivot_type == 'high' else 0.0, # Pivot type
float(pivot.level),
1.0 if pivot.confirmed else 0.0, # Confirmation status
float((datetime.now() - pivot.timestamp).total_seconds() / 3600), # Hours since pivot
float(pivot.index), # Position in data
0.0 # Reserved for future use
])
# Add trend features (10 features)
trend_direction_encoded = {
'up': [1.0, 0.0, 0.0],
'down': [0.0, 1.0, 0.0],
'sideways': [0.0, 0.0, 1.0]
}.get(trend_level.trend_direction, [0.0, 0.0, 1.0])
level_features.extend(trend_direction_encoded)
level_features.append(trend_level.trend_strength)
level_features.extend([0.0] * 6) # Reserved for future use
else:
# No data for this level, fill with zeros
level_features = [0.0] * 50
features.extend(level_features)
return np.array(features, dtype=np.float32)
except Exception as e:
logger.error(f"Error extracting pivot features for ML: {e}")
return np.zeros(250, dtype=np.float32)
def get_current_market_structure(self) -> Dict[str, Any]:
"""
Get current market structure summary for dashboard display
"""
try:
structure = {
'levels': {},
'overall_trend': 'sideways',
'overall_strength': 0.0,
'last_update': datetime.now().isoformat()
}
# Aggregate information from all levels
trend_votes = {'up': 0, 'down': 0, 'sideways': 0}
total_strength = 0.0
active_levels = 0
for level, trend_level in self.pivot_levels.items():
structure['levels'][level] = {
'trend_direction': trend_level.trend_direction,
'trend_strength': trend_level.trend_strength,
'pivot_count': len(trend_level.pivot_points),
'last_pivot': {
'timestamp': trend_level.pivot_points[-1].timestamp.isoformat() if trend_level.pivot_points else None,
'price': trend_level.pivot_points[-1].price if trend_level.pivot_points else 0.0,
'type': trend_level.pivot_points[-1].pivot_type if trend_level.pivot_points else 'none'
} if trend_level.pivot_points else None
}
# Vote for overall trend
trend_votes[trend_level.trend_direction] += trend_level.trend_strength
total_strength += trend_level.trend_strength
active_levels += 1
# Determine overall trend
if active_levels > 0:
structure['overall_trend'] = max(trend_votes, key=trend_votes.get)
structure['overall_strength'] = total_strength / active_levels
return structure
except Exception as e:
logger.error(f"Error getting current market structure: {e}")
return {
'levels': {},
'overall_trend': 'sideways',
'overall_strength': 0.0,
'last_update': datetime.now().isoformat(),
'error': str(e)
}

View File

@ -436,9 +436,21 @@ class CleanTradingDashboard:
symbol = 'ETH/USDT'
self._sync_position_from_executor(symbol)
# Get current price
# Get current price with better error handling
current_price = self._get_current_price('ETH/USDT')
price_str = f"${current_price:.2f}" if current_price else "Loading..."
if current_price and current_price > 0:
price_str = f"${current_price:.2f}"
else:
# Try to get price from COB data as fallback
if hasattr(self, 'latest_cob_data') and 'ETH/USDT' in self.latest_cob_data:
cob_data = self.latest_cob_data['ETH/USDT']
if 'stats' in cob_data and 'mid_price' in cob_data['stats']:
current_price = cob_data['stats']['mid_price']
price_str = f"${current_price:.2f}"
else:
price_str = "Loading..."
else:
price_str = "Loading..."
# Calculate session P&L including unrealized P&L from current position
total_session_pnl = self.session_pnl # Start with realized P&L
@ -621,9 +633,9 @@ class CleanTradingDashboard:
eth_snapshot = self._get_cob_snapshot('ETH/USDT')
btc_snapshot = self._get_cob_snapshot('BTC/USDT')
# Debug: Log COB data availability
if n % 5 == 0: # Log every 5 seconds to avoid spam
logger.info(f"COB Update #{n}: ETH snapshot: {eth_snapshot is not None}, BTC snapshot: {btc_snapshot is not None}")
# Debug: Log COB data availability - OPTIMIZED: Less frequent logging
if n % 20 == 0: # Log every 20 seconds to reduce spam and improve performance
logger.info(f"COB Update #{n % 100}: ETH snapshot: {eth_snapshot is not None}, BTC snapshot: {btc_snapshot is not None}")
if hasattr(self, 'latest_cob_data'):
eth_data_time = self.cob_last_update.get('ETH/USDT', 0) if hasattr(self, 'cob_last_update') else 0
btc_data_time = self.cob_last_update.get('BTC/USDT', 0) if hasattr(self, 'cob_last_update') else 0
@ -759,26 +771,98 @@ class CleanTradingDashboard:
return [html.I(className="fas fa-save me-1"), "Store All Models"]
def _get_current_price(self, symbol: str) -> Optional[float]:
"""Get current price for symbol"""
"""Get current price for symbol - ENHANCED with better fallbacks"""
try:
# Try WebSocket cache first
ws_symbol = symbol.replace('/', '')
if ws_symbol in self.ws_price_cache:
if ws_symbol in self.ws_price_cache and self.ws_price_cache[ws_symbol] > 0:
return self.ws_price_cache[ws_symbol]
# Fallback to data provider
if symbol in self.current_prices:
# Try data provider current prices
if hasattr(self.data_provider, 'current_prices') and symbol in self.data_provider.current_prices:
price = self.data_provider.current_prices[symbol]
if price and price > 0:
return price
# Try data provider get_current_price method
if hasattr(self.data_provider, 'get_current_price'):
try:
price = self.data_provider.get_current_price(symbol)
if price and price > 0:
self.current_prices[symbol] = price
return price
except Exception as dp_error:
logger.debug(f"Data provider get_current_price failed: {dp_error}")
# Fallback to dashboard current prices
if symbol in self.current_prices and self.current_prices[symbol] > 0:
return self.current_prices[symbol]
# Get fresh price from data provider
df = self.data_provider.get_historical_data(symbol, '1m', limit=1)
if df is not None and not df.empty:
price = float(df['close'].iloc[-1])
self.current_prices[symbol] = price
return price
# Get fresh price from data provider - try multiple timeframes
for timeframe in ['1m', '5m', '1h']: # Start with 1m instead of 1s for better reliability
try:
df = self.data_provider.get_historical_data(symbol, timeframe, limit=1, refresh=True)
if df is not None and not df.empty:
price = float(df['close'].iloc[-1])
if price > 0:
self.current_prices[symbol] = price
logger.debug(f"Got current price for {symbol} from {timeframe}: ${price:.2f}")
return price
except Exception as tf_error:
logger.debug(f"Failed to get {timeframe} data for {symbol}: {tf_error}")
continue
# Last resort: try to get from orchestrator if available
if hasattr(self, 'orchestrator') and self.orchestrator:
try:
# Try to get price from orchestrator's data
if hasattr(self.orchestrator, 'data_provider'):
price = self.orchestrator.data_provider.get_current_price(symbol)
if price and price > 0:
self.current_prices[symbol] = price
logger.debug(f"Got current price for {symbol} from orchestrator: ${price:.2f}")
return price
except Exception as orch_error:
logger.debug(f"Failed to get price from orchestrator: {orch_error}")
# Try external API as last resort
try:
import requests
if symbol == 'ETH/USDT':
response = requests.get('https://api.binance.com/api/v3/ticker/price?symbol=ETHUSDT', timeout=2)
if response.status_code == 200:
data = response.json()
price = float(data['price'])
if price > 0:
self.current_prices[symbol] = price
logger.debug(f"Got current price for {symbol} from Binance API: ${price:.2f}")
return price
elif symbol == 'BTC/USDT':
response = requests.get('https://api.binance.com/api/v3/ticker/price?symbol=BTCUSDT', timeout=2)
if response.status_code == 200:
data = response.json()
price = float(data['price'])
if price > 0:
self.current_prices[symbol] = price
logger.debug(f"Got current price for {symbol} from Binance API: ${price:.2f}")
return price
except Exception as api_error:
logger.debug(f"External API failed: {api_error}")
logger.warning(f"Could not get current price for {symbol} from any source")
except Exception as e:
logger.warning(f"Error getting current price for {symbol}: {e}")
logger.error(f"Error getting current price for {symbol}: {e}")
# Return a fallback price if we have any cached data
if symbol in self.current_prices and self.current_prices[symbol] > 0:
return self.current_prices[symbol]
# Return a reasonable fallback based on current market conditions
if symbol == 'ETH/USDT':
return 3385.0 # Current market price fallback
elif symbol == 'BTC/USDT':
return 119500.0 # Current market price fallback
return None