popov/gogo2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

remove emojis from console

Browse Source
This commit is contained in:
Dobromir Popov
2025-10-25 16:35:08 +03:00
parent 5aa4925cff
commit b8f54e61fa
75 changed files with 828 additions and 828 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
.github/workflows/ci-cd.yml vendored
View File

@@ -164,5 +164,5 @@ jobs:
- name: Notify on failure
if: ${{ needs.build-and-deploy.result == 'failure' || needs.docker-build.result == 'failure' }}
run: |
echo "❌ Deployment failed!"
echo " Deployment failed!"
# Add notification logic here (Slack, email, etc.)

38
.kiro/specs/1.multi-modal-trading-system/AUDIT_SUMMARY.md
View File

@@ -13,19 +13,19 @@ Comprehensive audit of the multi-modal trading system revealed a **strong, well-
┌─────────────────────────────────────────────────────────────┐
│ COBY System (Standalone) │
│ Multi-Exchange Aggregation │ TimescaleDB │ Redis Cache │
│ Status: ✅ Fully Operational │
│ Status: Fully Operational │
└─────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────┐
│ Core DataProvider (core/data_provider.py) │
│ Automatic Maintenance │ Williams Pivots │ COB Integration │
│ Status: ✅ Implemented, Needs Enhancement │
│ Status: Implemented, Needs Enhancement │
└─────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────┐
│ StandardizedDataProvider (core/standardized_data_provider.py) │
│ BaseDataInput │ ModelOutputManager │ Unified Interface │
│ Status: ✅ Implemented, Needs Heatmap Integration │
│ Status: Implemented, Needs Heatmap Integration │
└─────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────┐
@@ -35,7 +35,7 @@ Comprehensive audit of the multi-modal trading system revealed a **strong, well-
## Key Findings
### ✅ Strengths (Fully Implemented)
### Strengths (Fully Implemented)
1. **COBY System**
- Standalone multi-exchange data aggregation
@@ -82,7 +82,7 @@ Comprehensive audit of the multi-modal trading system revealed a **strong, well-
- Live price fetching with multiple fallbacks
- **Status**: Core functionality complete
### ⚠️ Partial Implementations (Needs Validation)
### Partial Implementations (Needs Validation)
1. **COB Raw Tick Storage**
- Structure exists (30 min buffer)
@@ -99,7 +99,7 @@ Comprehensive audit of the multi-modal trading system revealed a **strong, well-
- No unified interface yet
- Needs adapter layer
### ❌ Areas Needing Enhancement
### Areas Needing Enhancement
1. **COB Data Collection Robustness**
- **Issue**: NoneType errors in `_cob_aggregation_worker`
@@ -197,23 +197,23 @@ Dashboard / External Consumers
## Code Quality Assessment
### Excellent
- ✅ Comprehensive error handling in EnhancedCOBWebSocket
- ✅ Thread-safe data access patterns
- ✅ Clear separation of concerns across layers
- ✅ Extensive logging for debugging
- ✅ Proper use of dataclasses for type safety
- Comprehensive error handling in EnhancedCOBWebSocket
- Thread-safe data access patterns
- Clear separation of concerns across layers
- Extensive logging for debugging
- Proper use of dataclasses for type safety
### Good
- ✅ Automatic data maintenance workers
- ✅ Fallback mechanisms for API failures
- ✅ Subscriber pattern for data distribution
- ✅ Pivot-based normalization system
- Automatic data maintenance workers
- Fallback mechanisms for API failures
- Subscriber pattern for data distribution
- Pivot-based normalization system
### Needs Improvement
- ⚠️ Defensive programming in COB aggregation
- ⚠️ Configuration management (hardcoded values)
- ⚠️ Comprehensive input validation
- ⚠️ Data quality monitoring
- Defensive programming in COB aggregation
- Configuration management (hardcoded values)
- Comprehensive input validation
- Data quality monitoring
## Recommendations

32
.kiro/specs/1.multi-modal-trading-system/design.md
View File

@@ -83,7 +83,7 @@ The Data Provider backbone is the foundation of the system, implemented as a mul
- `HeatmapData`: Visualization-ready heatmap data
- `ConnectionStatus`: Exchange connection monitoring
**Current Status**: ✅ Fully implemented and operational
**Current Status**: Fully implemented and operational
#### Layer 2: Core DataProvider (Real-Time Trading Operations)
@@ -130,7 +130,7 @@ The Data Provider backbone is the foundation of the system, implemented as a mul
- `DataSubscriber`: Subscriber information
- `SimplePivotLevel`: Fallback pivot structure
**Current Status**: ✅ Fully implemented with ongoing enhancements
**Current Status**: Fully implemented with ongoing enhancements
#### Layer 3: StandardizedDataProvider (Unified Model Interface)
@@ -181,26 +181,26 @@ The Data Provider backbone is the foundation of the system, implemented as a mul
- Data completeness scoring
- Validation before model inference
**Current Status**: ✅ Implemented with enhancements needed for heatmap integration
**Current Status**: Implemented with enhancements needed for heatmap integration
#### Implementation Details
**Existing Strengths**:
- ✅ Robust automatic data maintenance with background workers
- ✅ Williams Market Structure with 5-level pivot analysis
- ✅ Real-time COB streaming with multiple Binance streams
- ✅ Thread-safe data access and subscriber management
- ✅ Comprehensive error handling and fallback mechanisms
- ✅ Pivot-based normalization for improved model training
- ✅ Centralized model output storage for cross-feeding
- Robust automatic data maintenance with background workers
- Williams Market Structure with 5-level pivot analysis
- Real-time COB streaming with multiple Binance streams
- Thread-safe data access and subscriber management
- Comprehensive error handling and fallback mechanisms
- Pivot-based normalization for improved model training
- Centralized model output storage for cross-feeding
**Areas for Enhancement**:
- ❌ Unified integration between COBY and core DataProvider
- ❌ COB heatmap matrix generation for model inputs
- ❌ Configurable price ranges for COB imbalance calculation
- ❌ Comprehensive data quality scoring and monitoring
- ❌ Missing data interpolation strategies
- ❌ Enhanced validation with detailed error reporting
- Unified integration between COBY and core DataProvider
- COB heatmap matrix generation for model inputs
- Configurable price ranges for COB imbalance calculation
- Comprehensive data quality scoring and monitoring
- Missing data interpolation strategies
- Enhanced validation with detailed error reporting
### Standardized Model Input/Output Format

54
.kiro/specs/1.multi-modal-trading-system/requirements.md
View File

@@ -23,26 +23,26 @@ The system is designed to adapt to current market conditions through continuous
#### Current Implementation Status
**IMPLEMENTED:**
- ✅ Core DataProvider with automatic data maintenance (1500 candles cached per symbol/timeframe)
- ✅ Multi-exchange COB integration via EnhancedCOBWebSocket (Binance depth@100ms, ticker, aggTrade streams)
- ✅ Williams Market Structure pivot point calculation with monthly data analysis
- ✅ Pivot-based normalization system with PivotBounds caching
- ✅ Real-time tick aggregation with RealTimeTickAggregator
- ✅ COB 1s aggregation with price buckets ($1 for ETH, $10 for BTC)
- ✅ Multi-timeframe imbalance calculations (1s, 5s, 15s, 60s MA)
- ✅ Centralized data distribution with subscriber management
- ✅ COBY standalone system with TimescaleDB storage and Redis caching
- Core DataProvider with automatic data maintenance (1500 candles cached per symbol/timeframe)
- Multi-exchange COB integration via EnhancedCOBWebSocket (Binance depth@100ms, ticker, aggTrade streams)
- Williams Market Structure pivot point calculation with monthly data analysis
- Pivot-based normalization system with PivotBounds caching
- Real-time tick aggregation with RealTimeTickAggregator
- COB 1s aggregation with price buckets ($1 for ETH, $10 for BTC)
- Multi-timeframe imbalance calculations (1s, 5s, 15s, 60s MA)
- Centralized data distribution with subscriber management
- COBY standalone system with TimescaleDB storage and Redis caching
**PARTIALLY IMPLEMENTED:**
- ⚠️ COB raw tick storage (30 min buffer) - implemented but needs validation
- ⚠️ Training data collection callbacks - structure exists but needs integration
- ⚠️ Cross-exchange COB consolidation - COBY system separate from core
- COB raw tick storage (30 min buffer) - implemented but needs validation
- Training data collection callbacks - structure exists but needs integration
- Cross-exchange COB consolidation - COBY system separate from core
**NEEDS ENHANCEMENT:**
- ❌ Unified integration between COBY and core DataProvider
- ❌ Configurable price range for COB imbalance (currently hardcoded $5 ETH, $50 BTC)
- ❌ COB heatmap matrix generation for model inputs
- ❌ Validation of 600-bar caching for backtesting support
- Unified integration between COBY and core DataProvider
- Configurable price range for COB imbalance (currently hardcoded $5 ETH, $50 BTC)
- COB heatmap matrix generation for model inputs
- Validation of 600-bar caching for backtesting support
#### Acceptance Criteria
@@ -98,19 +98,19 @@ The system is designed to adapt to current market conditions through continuous
#### Current Implementation Status
**IMPLEMENTED:**
- ✅ StandardizedDataProvider extending core DataProvider
- ✅ BaseDataInput dataclass with comprehensive fields
- ✅ OHLCVBar, COBData, PivotPoint, ModelOutput dataclasses
- ✅ ModelOutputManager for extensible cross-model feeding
- ✅ COB moving average calculation with thread-safe access
- ✅ Input validation before model inference
- ✅ Live price fetching with multiple fallbacks
- StandardizedDataProvider extending core DataProvider
- BaseDataInput dataclass with comprehensive fields
- OHLCVBar, COBData, PivotPoint, ModelOutput dataclasses
- ModelOutputManager for extensible cross-model feeding
- COB moving average calculation with thread-safe access
- Input validation before model inference
- Live price fetching with multiple fallbacks
**NEEDS ENHANCEMENT:**
- ❌ COB heatmap matrix integration in BaseDataInput
- ❌ Comprehensive data completeness validation
- ❌ Automatic data quality scoring
- ❌ Missing data interpolation strategies
- COB heatmap matrix integration in BaseDataInput
- Comprehensive data completeness validation
- Automatic data quality scoring
- Missing data interpolation strategies
#### Acceptance Criteria

4
.vscode/launch.json vendored
View File

@@ -135,7 +135,7 @@
"preLaunchTask": "Kill Stale Processes"
},
{
"name": "🚀 Integrated COB Dashboard + RL Trading",
"name": " Integrated COB Dashboard + RL Trading",
"type": "python",
"request": "launch",
"program": "run_integrated_rl_cob_dashboard.py",
@@ -202,7 +202,7 @@
],
"compounds": [
{
"name": "🚀 Full System (Dashboard + Training)",
"name": " Full System (Dashboard + Training)",
"configurations": [
"📊 Dashboard (Real-time + Training)",
"📊 TensorBoard Monitor"

54
ANNOTATE/IMPLEMENTATION_SUMMARY.md
View File

@@ -4,22 +4,22 @@
The Manual Trade Annotation UI is now **functionally complete** with all core features implemented and ready for use.
## ✅ Completed Tasks (Tasks 1-5)
## Completed Tasks (Tasks 1-5)
### Task 1: Project Structure ✅
### Task 1: Project Structure
- Complete folder structure in `/ANNOTATE`
- Flask/Dash web application
- Template-based architecture (all HTML in separate files)
- Dark theme CSS
- Client-side JavaScript modules
### Task 2: Data Loading ✅
### Task 2: Data Loading
- `HistoricalDataLoader` - Integrates with existing DataProvider
- `TimeRangeManager` - Time navigation and prefetching
- Memory caching with TTL
- **Uses same data source as training/inference**
### Task 3: Chart Visualization ✅
### Task 3: Chart Visualization
- Multi-timeframe Plotly charts (1s, 1m, 1h, 1d)
- Candlestick + volume visualization
- Chart synchronization across timeframes
@@ -27,14 +27,14 @@ The Manual Trade Annotation UI is now **functionally complete** with all core fe
- Zoom and pan functionality
- Scroll zoom enabled
### Task 4: Time Navigation ✅
### Task 4: Time Navigation
- Date/time picker
- Quick range buttons (1h, 4h, 1d, 1w)
- Forward/backward navigation
- Keyboard shortcuts (arrow keys)
- Time range calculations
### Task 5: Trade Annotation ✅
### Task 5: Trade Annotation
- Click to mark entry/exit points
- Visual markers on charts (▲ entry, ▼ exit)
- P&L calculation and display
@@ -44,7 +44,7 @@ The Manual Trade Annotation UI is now **functionally complete** with all core fe
## 🎯 Key Features
### 1. Data Consistency ✅
### 1. Data Consistency
```python
# Same DataProvider used everywhere
DataProvider → HistoricalDataLoader → Annotation UI
@@ -52,7 +52,7 @@ DataProvider → HistoricalDataLoader → Annotation UI
Training/Inference
```
### 2. Test Case Generation ✅
### 2. Test Case Generation
```python
# Generates test cases in realtime format
{
@@ -75,14 +75,14 @@ DataProvider → HistoricalDataLoader → Annotation UI
}
```
### 3. Visual Annotation System ✅
### 3. Visual Annotation System
- **Entry markers**: Green/Red triangles (▲)
- **Exit markers**: Green/Red triangles (▼)
- **P&L labels**: Displayed with percentage
- **Connecting lines**: Dashed lines between entry/exit
- **Color coding**: Green for LONG, Red for SHORT
### 4. Chart Features ✅
### 4. Chart Features
- **Multi-timeframe**: 4 synchronized charts
- **Candlestick**: OHLC visualization
- **Volume bars**: Color-coded by direction
@@ -164,7 +164,7 @@ Save to test_cases/annotation_*.json
└─────────────────────────────────────┘
```
## 🚀 Usage Guide
## Usage Guide
### 1. Start the Application
```bash
@@ -278,12 +278,12 @@ Export annotations to JSON/CSV
## 🎯 Next Steps (Optional Enhancements)
### Task 6: Annotation Storage ✅ (Already Complete)
### Task 6: Annotation Storage (Already Complete)
- JSON-based storage implemented
- CRUD operations working
- Auto-save functionality
### Task 7: Test Case Generation ✅ (Already Complete)
### Task 7: Test Case Generation (Already Complete)
- Realtime format implemented
- Market context extraction working
- File storage implemented
@@ -304,14 +304,14 @@ Export annotations to JSON/CSV
## ✨ Key Achievements
1. **✅ Data Consistency**: Uses same DataProvider as training/inference
2. **✅ Template Architecture**: All HTML in separate files
3. **✅ Dark Theme**: Professional UI matching main dashboard
4. **✅ Multi-Timeframe**: 4 synchronized charts
5. **✅ Visual Annotations**: Clear entry/exit markers with P&L
6. **✅ Test Case Generation**: Realtime format with market context
7. **✅ Self-Contained**: Isolated in /ANNOTATE folder
8. **✅ Production Ready**: Functional core features complete
1. ** Data Consistency**: Uses same DataProvider as training/inference
2. ** Template Architecture**: All HTML in separate files
3. ** Dark Theme**: Professional UI matching main dashboard
4. ** Multi-Timeframe**: 4 synchronized charts
5. ** Visual Annotations**: Clear entry/exit markers with P&L
6. ** Test Case Generation**: Realtime format with market context
7. ** Self-Contained**: Isolated in /ANNOTATE folder
8. ** Production Ready**: Functional core features complete
## 🎊 Success Criteria Met
@@ -326,14 +326,14 @@ Export annotations to JSON/CSV
- [ ] Model training integration (optional)
- [ ] Inference simulation (optional)
## 🚀 Ready for Use!
## Ready for Use!
The ANNOTATE system is now **ready for production use**. You can:
1. ✅ Mark profitable trades on historical data
2. ✅ Generate training test cases
3. ✅ Visualize annotations on charts
4. ✅ Export annotations for analysis
5. ✅ Use same data as training/inference
1. Mark profitable trades on historical data
2. Generate training test cases
3. Visualize annotations on charts
4. Export annotations for analysis
5. Use same data as training/inference
The core functionality is complete and the system is ready to generate high-quality training data for your models! 🎉

52
ANNOTATE/PROGRESS.md
View File

@@ -1,8 +1,8 @@
# ANNOTATE Project Progress
## ✅ Completed Tasks
## Completed Tasks
### Task 1: Project Structure and Base Templates ✅
### Task 1: Project Structure and Base Templates
**Status**: Complete
**What was built**:
@@ -33,7 +33,7 @@ ANNOTATE/
└── data/ (storage directories)
```
### Task 2: Data Loading and Caching Layer ✅
### Task 2: Data Loading and Caching Layer
**Status**: Complete
**What was built**:
@@ -46,11 +46,11 @@ ANNOTATE/
- Prefetching for smooth scrolling
**Key Features**:
- ✅ Uses the **same DataProvider** as training/inference systems
- ✅ Ensures **data consistency** across annotation, training, and inference
- ✅ Caches data for performance
- ✅ Supports time-based navigation
- ✅ Prefetches adjacent ranges for smooth UX
- Uses the **same DataProvider** as training/inference systems
- Ensures **data consistency** across annotation, training, and inference
- Caches data for performance
- Supports time-based navigation
- Prefetches adjacent ranges for smooth UX
**Integration Points**:
```python
@@ -67,14 +67,14 @@ df = data_loader.get_data('ETH/USDT', '1m', limit=500)
## 🎯 Current Status
### Application Status
- ✅ Flask server running on http://127.0.0.1:8051
- ✅ Templates rendering correctly
- ✅ Data loading integrated with existing DataProvider
- ✅ Dark theme UI implemented
- ✅ Chart visualization (COMPLETE)
- ✅ Annotation functionality (COMPLETE)
- ✅ Test case generation (COMPLETE)
- ✅ **CORE FEATURES COMPLETE - READY FOR USE!**
- Flask server running on http://127.0.0.1:8051
- Templates rendering correctly
- Data loading integrated with existing DataProvider
- Dark theme UI implemented
- Chart visualization (COMPLETE)
- Annotation functionality (COMPLETE)
- Test case generation (COMPLETE)
- **CORE FEATURES COMPLETE - READY FOR USE!**
### Data Flow
```
@@ -146,11 +146,11 @@ The ANNOTATE system ensures data consistency by:
5. **Shared Configuration**: Uses main config.yaml
### Architecture Benefits
- ✅ **No Data Duplication**: Single source of truth
- ✅ **Consistent Quality**: Same data cleaning/validation
- ✅ **Performance**: Leverages existing caching
- ✅ **Maintainability**: Changes to DataProvider automatically propagate
- ✅ **Testing**: Annotations use same data as models see
- **No Data Duplication**: Single source of truth
- **Consistent Quality**: Same data cleaning/validation
- **Performance**: Leverages existing caching
- **Maintainability**: Changes to DataProvider automatically propagate
- **Testing**: Annotations use same data as models see
### Test Case Generation
When an annotation is created, the system will:
@@ -162,7 +162,7 @@ When an annotation is created, the system will:
This ensures models can be trained on manually validated scenarios using the exact same data structure.
## 🚀 Running the Application
## Running the Application
### Start the Server
```bash
@@ -198,10 +198,10 @@ test_cases = annotation_mgr.get_test_cases()
## 📝 Notes
- All HTML is in templates (requirement met ✅)
- Dark theme implemented (requirement met ✅)
- Data consistency ensured (requirement met ✅)
- Self-contained in /ANNOTATE folder (requirement met ✅)
- All HTML is in templates (requirement met )
- Dark theme implemented (requirement met )
- Data consistency ensured (requirement met )
- Self-contained in /ANNOTATE folder (requirement met )
- Ready for chart implementation (next step)
## 🎯 Success Criteria

6
ANNOTATE/README.md
View File

@@ -4,7 +4,7 @@
A professional web-based interface for manually marking profitable buy/sell signals on historical market data to generate high-quality training test cases for machine learning models.
**Status**: ✅ **Production Ready** - Core features complete and tested
**Status**: **Production Ready** - Core features complete and tested
## ✨ Key Features
@@ -27,7 +27,7 @@ A professional web-based interface for manually marking profitable buy/sell sign
- **Data consistency**: Uses same DataProvider as training/inference
- **Auto-save**: Test cases saved to JSON files
### 🔄 Data Integration
### Data Integration
- **Existing DataProvider**: No duplicate data fetching
- **Cached data**: Leverages existing cache
- **Same quality**: Identical data structure as models see
@@ -39,7 +39,7 @@ A professional web-based interface for manually marking profitable buy/sell sign
- **Responsive**: Works on different screen sizes
- **Keyboard shortcuts**: Arrow keys for navigation
## 🚀 Quick Start
## Quick Start
### Installation

56
ANNOTATE/REALTIME_INFERENCE_GUIDE.md
View File

@@ -6,7 +6,7 @@ Real-time inference mode runs your trained model on **live streaming data** from
---
## 🚀 Starting Real-Time Inference
## Starting Real-Time Inference
### Step 1: Select Model
Choose the model you want to run from the dropdown in the training panel.
@@ -97,16 +97,16 @@ Charts updating every 1s
- **False positives** - Signals that shouldn't happen
### Good Signs
- ✅ Signals at key levels (support/resistance)
- ✅ High confidence (>70%)
- ✅ Signals match your analysis
- ✅ Few false positives
- Signals at key levels (support/resistance)
- High confidence (>70%)
- Signals match your analysis
- Few false positives
### Warning Signs
- ⚠️ Too many signals (every second)
- ⚠️ Low confidence (<50%)
- ⚠️ Random signals
- ⚠️ Signals don't match patterns
- Too many signals (every second)
- Low confidence (<50%)
- Random signals
- Signals don't match patterns
---
@@ -234,7 +234,7 @@ All 4 charts update simultaneously. Watch for:
- Signals match training patterns
- Timing is precise
- No false positives
- Model learned correctly ✅
- Model learned correctly
**5. Stop Inference**
- Click "Stop Inference"
@@ -247,34 +247,34 @@ All 4 charts update simultaneously. Watch for:
## 🎯 Best Practices
### Before Starting
- ✅ Train model first
- ✅ Verify model loaded
- ✅ Check DataProvider has data
- ✅ Close unnecessary tabs
- Train model first
- Verify model loaded
- Check DataProvider has data
- Close unnecessary tabs
### During Inference
- ✅ Monitor all timeframes
- ✅ Note signal quality
- ✅ Check confidence levels
- ✅ Compare with your analysis
- Monitor all timeframes
- Note signal quality
- Check confidence levels
- Compare with your analysis
### After Stopping
- ✅ Review signal history
- ✅ Note performance
- ✅ Identify improvements
- ✅ Adjust training if needed
- Review signal history
- Note performance
- Identify improvements
- Adjust training if needed
---
## 🚀 Summary
## Summary
Real-time inference provides:
✅ **Live chart updates** (1/second)
✅ **Model predictions** in real-time
✅ **Signal markers** on charts
✅ **Confidence levels** displayed
✅ **Performance monitoring** built-in
**Live chart updates** (1/second)
**Model predictions** in real-time
**Signal markers** on charts
**Confidence levels** displayed
**Performance monitoring** built-in
Use it to:
- **Validate training** - Check model learned correctly

100
ANNOTATE/STATUS.md
View File

@@ -3,62 +3,62 @@
## 🎉 Project Complete!
**Date**: January 2025
**Status**: ✅ **Production Ready**
**Status**: **Production Ready**
**Completion**: **Tasks 1-8 Complete** (Core + Model Integration)
---
## ✅ Completed Tasks Summary
## Completed Tasks Summary
### ✅ Task 1: Project Structure and Base Templates
### Task 1: Project Structure and Base Templates
- Complete folder structure in `/ANNOTATE`
- Flask/Dash application framework
- Template-based architecture (all HTML separate)
- Dark theme CSS styling
- Client-side JavaScript modules
### ✅ Task 2: Data Loading and Caching Layer
### Task 2: Data Loading and Caching Layer
- `HistoricalDataLoader` class
- `TimeRangeManager` for navigation
- Integration with existing DataProvider
- Memory caching with TTL
- Multi-timeframe data loading
### ✅ Task 3: Multi-Timeframe Chart Visualization
### Task 3: Multi-Timeframe Chart Visualization
- Plotly candlestick charts (4 timeframes)
- Volume bars with color coding
- Chart synchronization
- Hover information display
- Zoom and pan functionality
### ✅ Task 4: Time Navigation System
### Task 4: Time Navigation System
- Date/time picker
- Quick range buttons
- Forward/backward navigation
- Keyboard shortcuts
- Time range calculations
### ✅ Task 5: Trade Annotation System
### Task 5: Trade Annotation System
- Click-to-mark entry/exit
- Visual markers (▲▼)
- P&L calculation
- Connecting lines
- Edit/delete functionality
### ✅ Task 6: Annotation Storage and Management
### Task 6: Annotation Storage and Management
- JSON-based storage
- CRUD operations
- Annotation validation
- Listing UI
- Export functionality
### ✅ Task 7: Test Case Generation System
### Task 7: Test Case Generation System
- Realtime format generation
- Market context extraction
- File storage
- DataProvider integration
### ✅ Task 8: Model Loading and Management
### Task 8: Model Loading and Management
- TrainingSimulator class
- Model loading from orchestrator
- Available models API
@@ -76,51 +76,51 @@
- **Total Lines**: ~2,500+ lines of code
### Features Implemented
- ✅ Multi-timeframe charts (4 timeframes)
- ✅ Visual annotations with P&L
- ✅ Test case generation
- ✅ Data consistency with training
- ✅ Model integration
- ✅ Dark theme UI
- ✅ Keyboard shortcuts
- ✅ Export functionality
- Multi-timeframe charts (4 timeframes)
- Visual annotations with P&L
- Test case generation
- Data consistency with training
- Model integration
- Dark theme UI
- Keyboard shortcuts
- Export functionality
### API Endpoints
- ✅ `/` - Main dashboard
- ✅ `/api/chart-data` - Get chart data
- ✅ `/api/save-annotation` - Save annotation
- ✅ `/api/delete-annotation` - Delete annotation
- ✅ `/api/generate-test-case` - Generate test case
- ✅ `/api/export-annotations` - Export annotations
- ✅ `/api/train-model` - Start training
- ✅ `/api/training-progress` - Get progress
- ✅ `/api/available-models` - List models
- `/` - Main dashboard
- `/api/chart-data` - Get chart data
- `/api/save-annotation` - Save annotation
- `/api/delete-annotation` - Delete annotation
- `/api/generate-test-case` - Generate test case
- `/api/export-annotations` - Export annotations
- `/api/train-model` - Start training
- `/api/training-progress` - Get progress
- `/api/available-models` - List models
---
## 🎯 Key Achievements
### 1. Data Consistency ✅
### 1. Data Consistency
**Problem**: Annotations need same data as training/inference
**Solution**: Integrated with existing DataProvider
**Result**: Perfect data consistency across all systems
### 2. Visual Annotation System ✅
### 2. Visual Annotation System
**Problem**: Need intuitive way to mark trades
**Solution**: Click-based marking with visual feedback
**Result**: Professional TradingView-like interface
### 3. Test Case Generation ✅
### 3. Test Case Generation
**Problem**: Need training data in correct format
**Solution**: Generate test cases with full market context
**Result**: Ready-to-use training data
### 4. Model Integration ✅
### 4. Model Integration
**Problem**: Need to load and use existing models
**Solution**: TrainingSimulator with orchestrator integration
**Result**: Can load CNN, DQN, Transformer, COB models
### 5. Template Architecture ✅
### 5. Template Architecture
**Problem**: Maintainable HTML structure
**Solution**: Jinja2 templates with component separation
**Result**: Clean, maintainable codebase
@@ -237,7 +237,7 @@ User Click → JavaScript → Flask API → AnnotationManager → JSON Storage
---
## 🚀 Deployment Checklist
## Deployment Checklist
- [x] Code complete and tested
- [x] Documentation written
@@ -255,22 +255,22 @@ User Click → JavaScript → Flask API → AnnotationManager → JSON Storage
## 📊 Success Metrics
### Functionality
- ✅ 100% of core features implemented
- ✅ 100% of API endpoints working
- ✅ 100% data consistency achieved
- ✅ 100% template-based architecture
- 100% of core features implemented
- 100% of API endpoints working
- 100% data consistency achieved
- 100% template-based architecture
### Quality
- ✅ Clean code structure
- ✅ Comprehensive documentation
- ✅ Error handling
- ✅ Performance optimized
- Clean code structure
- Comprehensive documentation
- Error handling
- Performance optimized
### Integration
- ✅ DataProvider integration
- ✅ Orchestrator integration
- ✅ Model loading
- ✅ Test case generation
- DataProvider integration
- Orchestrator integration
- Model loading
- Test case generation
---
@@ -308,11 +308,11 @@ The ANNOTATE project is **complete and production-ready**. All core features hav
5. **Production Ready**: Fully functional and documented
### Ready For
- ✅ Marking profitable trades
- ✅ Generating training test cases
- ✅ Model training integration
- ✅ Production deployment
- ✅ Team usage
- Marking profitable trades
- Generating training test cases
- Model training integration
- Production deployment
- Team usage
**Status**: 🎉 **COMPLETE AND READY FOR USE!**

30
ANNOTATE/TRAINING_DATA_FORMAT.md
View File

@@ -3,9 +3,9 @@
## 🎯 Overview
The ANNOTATE system generates training data that includes **±5 minutes of market data** around each trade signal. This allows models to learn:
- ✅ **WHERE to generate signals** (at entry/exit points)
- ✅ **WHERE NOT to generate signals** (before entry, after exit)
- ✅ **Context around the signal** (what led to the trade)
- **WHERE to generate signals** (at entry/exit points)
- **WHERE NOT to generate signals** (before entry, after exit)
- **Context around the signal** (what led to the trade)
---
@@ -259,11 +259,11 @@ for timestamp, label in zip(timestamps, labels):
```
**Model Learns:**
- ✅ Don't signal during consolidation
- ✅ Signal at breakout confirmation
- ✅ Hold during profitable move
- ✅ Exit at target
- ✅ Don't signal after exit
- Don't signal during consolidation
- Signal at breakout confirmation
- Hold during profitable move
- Exit at target
- Don't signal after exit
---
@@ -290,16 +290,16 @@ print(f"EXIT: {labels.count(3)}")
---
## 🚀 Summary
## Summary
The ANNOTATE system generates **production-ready training data** with:
✅ **±5 minutes of context** around each signal
✅ **Training labels** for each timestamp
✅ **Negative examples** (where NOT to signal)
✅ **Positive examples** (where TO signal)
✅ **All 4 timeframes** (1s, 1m, 1h, 1d)
✅ **Complete market state** (OHLCV data)
**±5 minutes of context** around each signal
**Training labels** for each timestamp
**Negative examples** (where NOT to signal)
**Positive examples** (where TO signal)
**All 4 timeframes** (1s, 1m, 1h, 1d)
**Complete market state** (OHLCV data)
This enables models to learn:
- **Precise timing** of entry/exit signals

26
ANNOTATE/TRAINING_GUIDE.md
View File

@@ -18,11 +18,11 @@ When you save an annotation, a test case is **automatically generated** and save
### What's Included
Each test case contains:
- ✅ **Market State** - OHLCV data for all 4 timeframes (100 candles each)
- ✅ **Entry/Exit Prices** - Exact prices from annotation
- ✅ **Expected Outcome** - Direction (LONG/SHORT) and P&L percentage
- ✅ **Timestamp** - When the trade occurred
- ✅ **Action** - BUY or SELL signal
- **Market State** - OHLCV data for all 4 timeframes (100 candles each)
- **Entry/Exit Prices** - Exact prices from annotation
- **Expected Outcome** - Direction (LONG/SHORT) and P&L percentage
- **Timestamp** - When the trade occurred
- **Action** - BUY or SELL signal
### Test Case Format
```json
@@ -105,7 +105,7 @@ The system integrates with your existing models:
---
## 🚀 Real-Time Inference
## Real-Time Inference
### Overview
Real-time inference mode runs your trained model on **live streaming data** from the DataProvider, generating predictions in real-time.
@@ -298,7 +298,7 @@ model = orchestrator.cob_rl_agent
---
## 🚀 Advanced Usage
## Advanced Usage
### Custom Training Parameters
Edit `ANNOTATE/core/training_simulator.py`:
@@ -345,11 +345,11 @@ models/checkpoints/ (main system)
The ANNOTATE system provides:
✅ **Automatic Test Case Generation** - From annotations
✅ **Production-Ready Training** - Integrates with orchestrator
✅ **Real-Time Inference** - Live predictions on streaming data
✅ **Data Consistency** - Same data as main system
✅ **Easy Monitoring** - Real-time progress and signals
**Automatic Test Case Generation** - From annotations
**Production-Ready Training** - Integrates with orchestrator
**Real-Time Inference** - Live predictions on streaming data
**Data Consistency** - Same data as main system
**Easy Monitoring** - Real-time progress and signals
**You can now:**
1. Mark profitable trades
@@ -360,4 +360,4 @@ The ANNOTATE system provides:
---
**Happy Training!** 🚀
**Happy Training!**

32
ANNOTATE/USAGE_GUIDE.md
View File

@@ -24,12 +24,12 @@ Access at: **http://127.0.0.1:8051**
### What Gets Captured
When you create an annotation, the system captures:
- ✅ **Entry timestamp and price**
- ✅ **Exit timestamp and price**
- ✅ **Full market state** (OHLCV for all 4 timeframes)
- ✅ **Direction** (LONG/SHORT)
- ✅ **P&L percentage**
- ✅ **Market context** at both entry and exit
- **Entry timestamp and price**
- **Exit timestamp and price**
- **Full market state** (OHLCV for all 4 timeframes)
- **Direction** (LONG/SHORT)
- **P&L percentage**
- **Market context** at both entry and exit
This ensures the annotation contains **exactly the same data** your models will see during training!
@@ -99,10 +99,10 @@ This ensures the annotation contains **exactly the same data** your models will
### Automatic Generation
When you save an annotation, the system:
1. ✅ Captures market state at entry time
2. ✅ Captures market state at exit time
3. ✅ Stores OHLCV data for all timeframes
4. ✅ Calculates expected outcome (P&L, direction)
1. Captures market state at entry time
2. Captures market state at exit time
3. Stores OHLCV data for all timeframes
4. Calculates expected outcome (P&L, direction)
### Manual Generation
1. Find annotation in sidebar
@@ -232,11 +232,11 @@ Export annotations regularly to backup your work.
### Why It Matters
The annotation system uses the **same DataProvider** as your training and inference systems. This means:
✅ **Same data source**
✅ **Same data quality**
✅ **Same data structure**
✅ **Same timeframes**
✅ **Same caching**
**Same data source**
**Same data quality**
**Same data structure**
**Same timeframes**
**Same caching**
### What This Guarantees
When you train a model on annotated data:
@@ -282,7 +282,7 @@ ANNOTATE/data/annotations/export_<timestamp>.json
---
## 🚀 Next Steps
## Next Steps
After creating annotations:

18
ANNOTATE/core/NO_SIMULATION_POLICY.md
View File

@@ -54,18 +54,18 @@ from utils.checkpoint_manager import get_checkpoint_manager
## NEVER DO THIS
❌ Create files with "simulator", "simulation", "mock", "fake" in the name
❌ Use placeholder/dummy training loops
❌ Return fake metrics or results
❌ Skip actual model training
Create files with "simulator", "simulation", "mock", "fake" in the name
Use placeholder/dummy training loops
Return fake metrics or results
Skip actual model training
## ALWAYS DO THIS
✅ Use real model training methods
✅ Integrate with existing training systems
✅ Save real checkpoints
✅ Track real metrics
✅ Handle real data
Use real model training methods
Integrate with existing training systems
Save real checkpoints
Track real metrics
Handle real data
---

6
ANNOTATE/core/data_loader.py
View File

@@ -82,7 +82,7 @@ class HistoricalDataLoader:
# Use cached data if we have enough candles
if len(cached_df) >= min(limit, 100): # Use cached if we have at least 100 candles
elapsed_ms = (time.time() - start_time_ms) * 1000
logger.debug(f"🚀 DataProvider cache hit for {symbol} {timeframe} ({len(cached_df)} candles, {elapsed_ms:.1f}ms)")
logger.debug(f" DataProvider cache hit for {symbol} {timeframe} ({len(cached_df)} candles, {elapsed_ms:.1f}ms)")
# Filter by time range with direction support
filtered_df = self._filter_by_time_range(
@@ -177,7 +177,7 @@ class HistoricalDataLoader:
if df is not None and not df.empty:
elapsed_ms = (time.time() - start_time_ms) * 1000
logger.info(f"✅ DuckDB hit for {symbol} {timeframe} ({len(df)} candles, {elapsed_ms:.1f}ms)")
logger.info(f" DuckDB hit for {symbol} {timeframe} ({len(df)} candles, {elapsed_ms:.1f}ms)")
# Cache in memory
self.memory_cache[cache_key] = (df.copy(), datetime.now())
return df
@@ -346,7 +346,7 @@ class HistoricalDataLoader:
df = df.set_index('timestamp')
df = df.sort_index()
logger.info(f"✅ Fetched {len(df)} candles from Binance for {symbol} {timeframe}")
logger.info(f" Fetched {len(df)} candles from Binance for {symbol} {timeframe}")
return df
except Exception as e:

40
ANNOTATE/core/real_training_adapter.py
View File

@@ -171,23 +171,23 @@ class RealTrainingAdapter:
if not training_data:
raise Exception("No valid training data prepared from test cases")
logger.info(f"✅ Prepared {len(training_data)} training samples")
logger.info(f" Prepared {len(training_data)} training samples")
# Route to appropriate REAL training method
if model_name in ["CNN", "StandardizedCNN"]:
logger.info("🔄 Starting CNN training...")
logger.info(" Starting CNN training...")
self._train_cnn_real(session, training_data)
elif model_name == "DQN":
logger.info("🔄 Starting DQN training...")
logger.info(" Starting DQN training...")
self._train_dqn_real(session, training_data)
elif model_name == "Transformer":
logger.info("🔄 Starting Transformer training...")
logger.info(" Starting Transformer training...")
self._train_transformer_real(session, training_data)
elif model_name == "COB":
logger.info("🔄 Starting COB training...")
logger.info(" Starting COB training...")
self._train_cob_real(session, training_data)
elif model_name == "Extrema":
logger.info("🔄 Starting Extrema training...")
logger.info(" Starting Extrema training...")
self._train_extrema_real(session, training_data)
else:
raise Exception(f"Unknown model type: {model_name}")
@@ -196,12 +196,12 @@ class RealTrainingAdapter:
session.status = 'completed'
session.duration_seconds = time.time() - session.start_time
logger.info(f"✅ REAL training completed: {training_id} in {session.duration_seconds:.2f}s")
logger.info(f" REAL training completed: {training_id} in {session.duration_seconds:.2f}s")
logger.info(f" Final loss: {session.final_loss}")
logger.info(f" Accuracy: {session.accuracy}")
except Exception as e:
logger.error(f"❌ REAL training failed: {e}", exc_info=True)
logger.error(f" REAL training failed: {e}", exc_info=True)
session.status = 'failed'
session.error = str(e)
session.duration_seconds = time.time() - session.start_time
@@ -266,15 +266,15 @@ class RealTrainingAdapter:
'close': df['close'].tolist(),
'volume': df['volume'].tolist()
}
logger.debug(f" ✅ {timeframe}: {len(df)} candles")
logger.debug(f" {timeframe}: {len(df)} candles")
else:
logger.warning(f" ❌ {timeframe}: No data")
logger.warning(f" {timeframe}: No data")
if market_state['timeframes']:
logger.info(f" ✅ Fetched market state with {len(market_state['timeframes'])} timeframes")
logger.info(f" Fetched market state with {len(market_state['timeframes'])} timeframes")
return market_state
else:
logger.warning(f" ❌ No market data fetched")
logger.warning(f" No market data fetched")
return {}
except Exception as e:
@@ -309,7 +309,7 @@ class RealTrainingAdapter:
expected_outcome = test_case.get('expected_outcome', {})
if not expected_outcome:
logger.warning(f"⚠️ Skipping test case {test_case.get('test_case_id')}: missing expected_outcome")
logger.warning(f" Skipping test case {test_case.get('test_case_id')}: missing expected_outcome")
continue
# Check if market_state is provided, if not, fetch it dynamically
@@ -320,7 +320,7 @@ class RealTrainingAdapter:
market_state = self._fetch_market_state_for_test_case(test_case)
if not market_state:
logger.warning(f"⚠️ Skipping test case {test_case.get('test_case_id')}: could not fetch market state")
logger.warning(f" Skipping test case {test_case.get('test_case_id')}: could not fetch market state")
continue
logger.debug(f" Test case {i+1}: has_market_state={bool(market_state)}, has_expected_outcome={bool(expected_outcome)}")
@@ -339,7 +339,7 @@ class RealTrainingAdapter:
}
training_data.append(entry_sample)
logger.debug(f" ✅ Entry sample: {entry_sample['direction']} @ {entry_sample['entry_price']}")
logger.debug(f" Entry sample: {entry_sample['direction']} @ {entry_sample['entry_price']}")
# Create HOLD samples (every candle while position is open)
# This teaches the model to maintain the position until exit
@@ -367,7 +367,7 @@ class RealTrainingAdapter:
'repetitions': training_repetitions
}
training_data.append(exit_sample)
logger.debug(f" ✅ Exit sample @ {exit_sample['exit_price']} ({exit_sample['profit_loss_pct']:.2f}%)")
logger.debug(f" Exit sample @ {exit_sample['exit_price']} ({exit_sample['profit_loss_pct']:.2f}%)")
# Create NEGATIVE samples (where model should NOT trade)
# These are candles before and after the signal
@@ -382,14 +382,14 @@ class RealTrainingAdapter:
logger.debug(f" ➕ Added {len(negative_samples)} negative samples (±{negative_samples_window} candles)")
except Exception as e:
logger.error(f"❌ Error preparing test case {i+1}: {e}")
logger.error(f" Error preparing test case {i+1}: {e}")
total_entry = sum(1 for s in training_data if s.get('label') == 'ENTRY')
total_hold = sum(1 for s in training_data if s.get('label') == 'HOLD')
total_exit = sum(1 for s in training_data if s.get('label') == 'EXIT')
total_no_trade = sum(1 for s in training_data if s.get('label') == 'NO_TRADE')
logger.info(f"✅ Prepared {len(training_data)} training samples from {len(test_cases)} test cases")
logger.info(f" Prepared {len(training_data)} training samples from {len(test_cases)} test cases")
logger.info(f" ENTRY samples: {total_entry}")
logger.info(f" HOLD samples: {total_hold}")
logger.info(f" EXIT samples: {total_exit}")
@@ -399,7 +399,7 @@ class RealTrainingAdapter:
logger.info(f" Ratio: 1:{total_no_trade/total_entry:.1f} (entry:no_trade)")
if len(training_data) < len(test_cases):
logger.warning(f"⚠️ Skipped {len(test_cases) - len(training_data)} test cases due to missing data")
logger.warning(f" Skipped {len(test_cases) - len(training_data)} test cases due to missing data")
return training_data
@@ -1048,7 +1048,7 @@ class RealTrainingAdapter:
if not converted_batches:
raise Exception("No valid training batches after conversion")
logger.info(f" ✅ Converted {len(training_data)} samples to {len(converted_batches)} training batches")
logger.info(f" Converted {len(training_data)} samples to {len(converted_batches)} training batches")
# Train using train_step for each batch
for epoch in range(session.total_epochs):

52
ANNOTATE/web/app.py
View File

@@ -196,14 +196,14 @@ class AnnotationDashboard:
for attempt in range(max_retries):
try:
if attempt > 0:
logger.info(f"🔄 Retry attempt {attempt + 1}/{max_retries} for model loading...")
logger.info(f" Retry attempt {attempt + 1}/{max_retries} for model loading...")
time.sleep(retry_delay)
else:
logger.info("🔄 Starting async model loading...")
logger.info(" Starting async model loading...")
# Check if TradingOrchestrator is available
if not TradingOrchestrator:
logger.error("❌ TradingOrchestrator class not available")
logger.error(" TradingOrchestrator class not available")
self.models_loading = False
self.available_models = []
return
@@ -214,48 +214,48 @@ class AnnotationDashboard:
data_provider=self.data_provider,
enhanced_rl_training=True
)
logger.info(" ✅ Orchestrator created")
logger.info(" Orchestrator created")
# Initialize ML models
logger.info(" Initializing ML models...")
self.orchestrator._initialize_ml_models()
logger.info(" ✅ ML models initialized")
logger.info(" ML models initialized")
# Update training adapter with orchestrator
self.training_adapter.orchestrator = self.orchestrator
logger.info(" ✅ Training adapter updated")
logger.info(" Training adapter updated")
# Get available models from orchestrator
available = []
if hasattr(self.orchestrator, 'rl_agent') and self.orchestrator.rl_agent:
available.append('DQN')
logger.info(" ✅ DQN model available")
logger.info(" DQN model available")
if hasattr(self.orchestrator, 'cnn_model') and self.orchestrator.cnn_model:
available.append('CNN')
logger.info(" ✅ CNN model available")
logger.info(" CNN model available")
if hasattr(self.orchestrator, 'transformer_model') and self.orchestrator.transformer_model:
available.append('Transformer')
logger.info(" ✅ Transformer model available")
logger.info(" Transformer model available")
self.available_models = available
if available:
logger.info(f"✅ Models loaded successfully: {', '.join(available)}")
logger.info(f" Models loaded successfully: {', '.join(available)}")
else:
logger.warning("⚠️ No models were initialized (this might be normal if models aren't configured)")
logger.warning(" No models were initialized (this might be normal if models aren't configured)")
self.models_loading = False
logger.info("✅ Async model loading complete")
logger.info(" Async model loading complete")
return # Success - exit retry loop
except Exception as e:
logger.error(f"❌ Error loading models (attempt {attempt + 1}/{max_retries}): {e}")
logger.error(f" Error loading models (attempt {attempt + 1}/{max_retries}): {e}")
import traceback
logger.error(f"Traceback:\n{traceback.format_exc()}")
if attempt == max_retries - 1:
# Final attempt failed
logger.error(f"❌ Model loading failed after {max_retries} attempts")
logger.error(f" Model loading failed after {max_retries} attempts")
self.models_loading = False
self.available_models = []
else:
@@ -264,7 +264,7 @@ class AnnotationDashboard:
# Start loading in background thread
thread = threading.Thread(target=load_models, daemon=True, name="ModelLoader")
thread.start()
logger.info(f"🚀 Model loading started in background thread (ID: {thread.ident}, Name: {thread.name})")
logger.info(f" Model loading started in background thread (ID: {thread.ident}, Name: {thread.name})")
logger.info(" UI remains responsive while models load...")
logger.info(" Will retry up to 3 times if loading fails")
@@ -284,7 +284,7 @@ class AnnotationDashboard:
)
if success:
logger.info("✅ ANNOTATE: Unified storage enabled for real-time data")
logger.info(" ANNOTATE: Unified storage enabled for real-time data")
# Get statistics
stats = self.data_provider.get_unified_storage_stats()
@@ -293,7 +293,7 @@ class AnnotationDashboard:
logger.info(" Historical data access: <100ms")
logger.info(" Annotation data: Available at any timestamp")
else:
logger.warning("⚠️ ANNOTATE: Unified storage not available, using cached data only")
logger.warning(" ANNOTATE: Unified storage not available, using cached data only")
except Exception as e:
logger.warning(f"ANNOTATE: Could not enable unified storage: {e}")
@@ -312,7 +312,7 @@ class AnnotationDashboard:
# Wait for app to fully start
time.sleep(5)
logger.info("🔄 Starting one-time background data refresh (fetching only recent missing data)")
logger.info(" Starting one-time background data refresh (fetching only recent missing data)")
# Disable startup mode to fetch fresh data
self.data_loader.disable_startup_mode()
@@ -321,7 +321,7 @@ class AnnotationDashboard:
logger.info("Using on-demand refresh for recent data")
self.data_provider.refresh_data_on_demand()
logger.info("✅ One-time background data refresh completed")
logger.info(" One-time background data refresh completed")
except Exception as e:
logger.error(f"Error in background data refresh: {e}")
@@ -488,9 +488,9 @@ class AnnotationDashboard:
<h1 class="text-center">📝 ANNOTATE - Manual Trade Annotation UI</h1>
<div class="alert alert-info">
<h4>System Status</h4>
<p>✅ Annotation Manager: Active</p>
<p>⚠️ Data Provider: {'Available' if self.data_provider else 'Not Available (Standalone Mode)'}</p>
<p>⚠️ Trading Orchestrator: {'Available' if self.orchestrator else 'Not Available (Standalone Mode)'}</p>
<p> Annotation Manager: Active</p>
<p> Data Provider: {'Available' if self.data_provider else 'Not Available (Standalone Mode)'}</p>
<p> Trading Orchestrator: {'Available' if self.orchestrator else 'Not Available (Standalone Mode)'}</p>
</div>
<div class="row">
<div class="col-md-6">
@@ -537,7 +537,7 @@ class AnnotationDashboard:
'error': {'code': 'INVALID_REQUEST', 'message': 'Missing timeframe or timestamps'}
})
logger.info(f"🔄 Recalculating pivots for {symbol} {timeframe} with {len(timestamps)} candles")
logger.info(f" Recalculating pivots for {symbol} {timeframe} with {len(timestamps)} candles")
# Convert to DataFrame
df = pd.DataFrame({
@@ -552,7 +552,7 @@ class AnnotationDashboard:
# Recalculate pivot markers
pivot_markers = self._get_pivot_markers_for_timeframe(symbol, timeframe, df)
logger.info(f" ✅ Recalculated {len(pivot_markers)} pivot candles")
logger.info(f" Recalculated {len(pivot_markers)} pivot candles")
return jsonify({
'success': True,
@@ -614,7 +614,7 @@ class AnnotationDashboard:
)
if df is not None and not df.empty:
logger.info(f" ✅ {timeframe}: {len(df)} candles ({df.index[0]} to {df.index[-1]})")
logger.info(f" {timeframe}: {len(df)} candles ({df.index[0]} to {df.index[-1]})")
# Get pivot points for this timeframe
pivot_markers = self._get_pivot_markers_for_timeframe(symbol, timeframe, df)
@@ -630,7 +630,7 @@ class AnnotationDashboard:
'pivot_markers': pivot_markers # Optional: only present if pivots exist
}
else:
logger.warning(f" ❌ {timeframe}: No data returned")
logger.warning(f" {timeframe}: No data returned")
# Get pivot bounds for the symbol
pivot_bounds = None

8
ANNOTATE/web/static/js/chart_manager.js
View File

@@ -1215,10 +1215,10 @@ class ChartManager {
// Merge with existing data
this.mergeChartData(timeframe, newData, direction);
console.log(`✅ Loaded ${newData.timestamps.length} new candles for ${timeframe}`);
console.log(` Loaded ${newData.timestamps.length} new candles for ${timeframe}`);
window.showSuccess(`Loaded ${newData.timestamps.length} more candles`);
} else {
console.warn(`❌ No more data available for ${timeframe} ${direction}`);
console.warn(` No more data available for ${timeframe} ${direction}`);
console.warn('Full result:', result);
window.showWarning('No more historical data available');
}
@@ -1312,7 +1312,7 @@ class ChartManager {
*/
async recalculatePivots(timeframe, data) {
try {
console.log(`🔄 Recalculating pivots for ${timeframe} with ${data.timestamps.length} candles...`);
console.log(` Recalculating pivots for ${timeframe} with ${data.timestamps.length} candles...`);
const response = await fetch('/api/recalculate-pivots', {
method: 'POST',
@@ -1338,7 +1338,7 @@ class ChartManager {
const chart = this.charts[timeframe];
if (chart && chart.data) {
chart.data.pivot_markers = result.pivot_markers;
console.log(`✅ Pivots recalculated: ${Object.keys(result.pivot_markers).length} pivot candles`);
console.log(` Pivots recalculated: ${Object.keys(result.pivot_markers).length} pivot candles`);
// Redraw the chart with updated pivots
this.redrawChartWithPivots(timeframe, chart.data);

10
ANNOTATE/web/templates/components/training_panel.html
View File

@@ -113,7 +113,7 @@
if (data.loading) {
// Models still loading - show loading message and poll
modelSelect.innerHTML = '<option value="">🔄 Loading models...</option>';
modelSelect.innerHTML = '<option value=""> Loading models...</option>';
// Start polling if not already polling
if (!modelLoadingPollInterval) {
@@ -132,7 +132,7 @@
if (data.success && data.models.length > 0) {
// Show success notification
if (window.showSuccess) {
window.showSuccess(`✅ ${data.models.length} models loaded and ready for training`);
window.showSuccess(` ${data.models.length} models loaded and ready for training`);
}
data.models.forEach(model => {
@@ -142,7 +142,7 @@
modelSelect.appendChild(option);
});
console.log(`✅ Models loaded: ${data.models.join(', ')}`);
console.log(` Models loaded: ${data.models.join(', ')}`);
} else {
const option = document.createElement('option');
option.value = '';
@@ -157,12 +157,12 @@
// Don't stop polling on network errors - keep trying
if (!modelLoadingPollInterval) {
modelSelect.innerHTML = '<option value="">⚠️ Connection error, retrying...</option>';
modelSelect.innerHTML = '<option value=""> Connection error, retrying...</option>';
// Start polling to retry
modelLoadingPollInterval = setInterval(loadAvailableModels, 3000); // Poll every 3 seconds
} else {
// Already polling, just update the message
modelSelect.innerHTML = '<option value="">🔄 Retrying...</option>';
modelSelect.innerHTML = '<option value=""> Retrying...</option>';
}
});
}

32
CLEANUP_SUMMARY.md
View File

@@ -7,7 +7,7 @@
## Changes Made
### Phase 1: Removed All Mock/Synthetic Data ✅
### Phase 1: Removed All Mock/Synthetic Data
**Policy Enforcement**:
- Added "NO SYNTHETIC DATA" policy warnings to all core modules
@@ -32,7 +32,7 @@
---
### Phase 2: Removed Unused Dashboard Implementations ✅
### Phase 2: Removed Unused Dashboard Implementations
**Files Deleted**:
- `web/templated_dashboard.py` (1000+ lines)
@@ -49,7 +49,7 @@
---
### Phase 3: Consolidated Training Runners ✅
### Phase 3: Consolidated Training Runners
**NEW FILE CREATED**:
- `training_runner.py` - Unified training system supporting:
@@ -71,7 +71,7 @@
---
### Phase 4: Consolidated Main Entry Points ✅
### Phase 4: Consolidated Main Entry Points
**NEW FILES CREATED**:
1. `main_dashboard.py` - Real-time dashboard & live training
@@ -95,7 +95,7 @@
---
### Phase 5: Fixed Broken Imports & Removed Unused Files ✅
### Phase 5: Fixed Broken Imports & Removed Unused Files
**Files Deleted**:
1. `tests/test_training_status.py` - Broken import (web.old_archived)
@@ -121,7 +121,7 @@
---
### Phase 6: Removed Unused Components ✅
### Phase 6: Removed Unused Components
**Files Deleted**:
- `NN/training/integrate_checkpoint_management.py` - Redundant with model_manager.py
@@ -134,7 +134,7 @@
---
### Phase 7: Documentation Updated ✅
### Phase 7: Documentation Updated
**Files Modified**:
- `readme.md` - Updated Quick Start section with new entry points
@@ -200,12 +200,12 @@ python training_runner.py --mode [realtime|backtest]
## Key Improvements
✅ **ZERO Mock/Synthetic Data** - All synthetic data generation removed
✅ **Single Training System** - 6 duplicate runners → 1 unified
✅ **Clear Entry Points** - 5 entry points → 2 focused
✅ **Cleaner Codebase** - 40+ unnecessary files removed
✅ **Better Maintainability** - Less duplication, clearer structure
✅ **No Broken Imports** - All dead code references removed
**ZERO Mock/Synthetic Data** - All synthetic data generation removed
**Single Training System** - 6 duplicate runners → 1 unified
**Clear Entry Points** - 5 entry points → 2 focused
**Cleaner Codebase** - 40+ unnecessary files removed
**Better Maintainability** - Less duplication, clearer structure
**No Broken Imports** - All dead code references removed
---
@@ -268,9 +268,9 @@ python main_backtest.py --start 2024-01-01 --end 2024-12-31
## Next Steps
1. ✅ Test `main_dashboard.py` for basic functionality
2. ✅ Test `main_backtest.py` with small date range
3. ✅ Test `training_runner.py` in both modes
1. Test `main_dashboard.py` for basic functionality
2. Test `main_backtest.py` with small date range
3. Test `training_runner.py` in both modes
4. Update `.vscode/launch.json` configurations
5. Run integration tests
6. Update any remaining documentation

4
COBY/README.md
View File

@@ -27,7 +27,7 @@ if os.path.exists(static_path):
app.mount("/static", StaticFiles(directory=static_path), name="static")
# Serve index.html at root for dashboard
app.mount("/", StaticFiles(directory=static_path, html=True), name="dashboard")
🚀 To access the dashboard:
To access the dashboard:
Start the application: python COBY/main.py --debug
Open browser: Navigate to http://localhost:8080/
API health check: http://localhost:8080/health
@@ -77,7 +77,7 @@ COBY/
└── README.md
```
## 🚀 Features
## Features
- **Multi-Exchange Support**: Connect to 10+ major cryptocurrency exchanges
- **Real-Time Data**: High-frequency order book and trade data collection

4
COBY/docker/README.md
View File

@@ -15,7 +15,7 @@ This directory contains Docker Compose configurations and scripts for deploying
- At least 4GB RAM available for containers
- 50GB+ disk space for data storage
## 🚀 Quick Start
## Quick Start
1. **Copy environment file**:
```bash
@@ -264,7 +264,7 @@ For issues related to:
- Redis: Check [Redis docs](https://redis.io/documentation)
- Docker: Check [Docker docs](https://docs.docker.com/)
## 🔄 Updates
## Updates
This infrastructure supports:
- Rolling updates with zero downtime

14
COBY/docker/backup.sh
View File

@@ -32,9 +32,9 @@ docker exec market_data_timescaledb pg_dump \
> "$BACKUP_DIR/timescaledb_backup_$TIMESTAMP.dump"
if [ $? -eq 0 ]; then
echo "✅ TimescaleDB backup completed: timescaledb_backup_$TIMESTAMP.dump"
echo " TimescaleDB backup completed: timescaledb_backup_$TIMESTAMP.dump"
else
echo "❌ TimescaleDB backup failed"
echo " TimescaleDB backup failed"
exit 1
fi
@@ -52,9 +52,9 @@ sleep 5
docker cp market_data_redis:/data/redis_backup_$TIMESTAMP.rdb "$BACKUP_DIR/"
if [ $? -eq 0 ]; then
echo "✅ Redis backup completed: redis_backup_$TIMESTAMP.rdb"
echo " Redis backup completed: redis_backup_$TIMESTAMP.rdb"
else
echo "❌ Redis backup failed"
echo " Redis backup failed"
exit 1
fi
@@ -88,7 +88,7 @@ rm "$BACKUP_DIR/timescaledb_backup_$TIMESTAMP.dump"
rm "$BACKUP_DIR/redis_backup_$TIMESTAMP.rdb"
rm "$BACKUP_DIR/backup_$TIMESTAMP.info"
echo "✅ Compressed backup created: market_data_backup_$TIMESTAMP.tar.gz"
echo " Compressed backup created: market_data_backup_$TIMESTAMP.tar.gz"
# Clean up old backups
echo "🧹 Cleaning up old backups (older than $RETENTION_DAYS days)..."
@@ -102,7 +102,7 @@ echo " File: market_data_backup_$TIMESTAMP.tar.gz"
echo " Size: $BACKUP_SIZE"
echo " Location: $BACKUP_DIR"
echo ""
echo "🔄 To restore from this backup:"
echo " To restore from this backup:"
echo " ./restore.sh market_data_backup_$TIMESTAMP.tar.gz"
echo ""
echo "✅ Backup process completed successfully!"
echo " Backup process completed successfully!"

8
COBY/docker/manual-init.sh
View File

@@ -14,7 +14,7 @@ echo "📡 Testing connection to TimescaleDB..."
PGPASSWORD="market_data_secure_pass_2024" psql -h 192.168.0.10 -p 5432 -U market_user -d market_data -c "SELECT version();"
if [ $? -eq 0 ]; then
echo "✅ Connection successful!"
echo " Connection successful!"
echo "🏗️ Creating database schema..."
@@ -22,16 +22,16 @@ if [ $? -eq 0 ]; then
PGPASSWORD="market_data_secure_pass_2024" psql -h 192.168.0.10 -p 5432 -U market_user -d market_data -f ../docker/init-scripts/01-init-timescaledb.sql
if [ $? -eq 0 ]; then
echo "✅ Database schema initialized successfully!"
echo " Database schema initialized successfully!"
echo "📊 Verifying tables..."
PGPASSWORD="market_data_secure_pass_2024" psql -h 192.168.0.10 -p 5432 -U market_user -d market_data -c "\dt market_data.*"
else
echo "❌ Schema initialization failed"
echo " Schema initialization failed"
exit 1
fi
else
echo "❌ Cannot connect to database"
echo " Cannot connect to database"
exit 1
fi

28
COBY/docker/restore.sh
View File

@@ -7,7 +7,7 @@ set -e
# Check if backup file is provided
if [ $# -eq 0 ]; then
echo "❌ Usage: $0 <backup_file.tar.gz>"
echo " Usage: $0 <backup_file.tar.gz>"
echo "Available backups:"
ls -la ./backups/market_data_backup_*.tar.gz 2>/dev/null || echo "No backups found"
exit 1
@@ -22,12 +22,12 @@ if [ -f .env ]; then
source .env
fi
echo "🔄 Starting restore process..."
echo " Starting restore process..."
echo "📁 Backup file: $BACKUP_FILE"
# Check if backup file exists
if [ ! -f "$BACKUP_FILE" ]; then
echo "❌ Backup file not found: $BACKUP_FILE"
echo " Backup file not found: $BACKUP_FILE"
exit 1
fi
@@ -44,7 +44,7 @@ REDIS_BACKUP=$(find "$RESTORE_DIR" -name "redis_backup_*.rdb" | head -1)
BACKUP_INFO=$(find "$RESTORE_DIR" -name "backup_*.info" | head -1)
if [ -z "$TIMESCALE_BACKUP" ] || [ -z "$REDIS_BACKUP" ]; then
echo "❌ Invalid backup file structure"
echo " Invalid backup file structure"
rm -rf "$RESTORE_DIR"
exit 1
fi
@@ -57,10 +57,10 @@ if [ -f "$BACKUP_INFO" ]; then
fi
# Confirm restore
read -p "⚠️ This will replace all existing data. Continue? (y/N): " -n 1 -r
read -p " This will replace all existing data. Continue? (y/N): " -n 1 -r
echo
if [[ ! $REPLY =~ ^[Yy]$ ]]; then
echo "❌ Restore cancelled"
echo " Restore cancelled"
rm -rf "$RESTORE_DIR"
exit 1
fi
@@ -85,7 +85,7 @@ sleep 30
# Check if TimescaleDB is ready
if ! docker exec market_data_timescaledb pg_isready -U market_user -d market_data; then
echo "❌ TimescaleDB is not ready"
echo " TimescaleDB is not ready"
exit 1
fi
@@ -105,9 +105,9 @@ docker exec market_data_timescaledb pg_restore \
/tmp/restore.dump
if [ $? -eq 0 ]; then
echo "✅ TimescaleDB restore completed"
echo " TimescaleDB restore completed"
else
echo "❌ TimescaleDB restore failed"
echo " TimescaleDB restore failed"
exit 1
fi
@@ -136,7 +136,7 @@ echo "🔍 Verifying restore..."
# Check TimescaleDB
if docker exec market_data_timescaledb pg_isready -U market_user -d market_data; then
echo "✅ TimescaleDB is ready"
echo " TimescaleDB is ready"
# Show table counts
echo "📊 Database table counts:"
@@ -150,19 +150,19 @@ if docker exec market_data_timescaledb pg_isready -U market_user -d market_data;
ORDER BY tablename;
"
else
echo "❌ TimescaleDB verification failed"
echo " TimescaleDB verification failed"
exit 1
fi
# Check Redis
if docker exec market_data_redis redis-cli -a "$REDIS_PASSWORD" ping | grep -q PONG; then
echo "✅ Redis is ready"
echo " Redis is ready"
# Show Redis info
echo "📦 Redis database info:"
docker exec market_data_redis redis-cli -a "$REDIS_PASSWORD" INFO keyspace
else
echo "❌ Redis verification failed"
echo " Redis verification failed"
exit 1
fi
@@ -178,7 +178,7 @@ echo " Source: $BACKUP_FILE"
echo " Timestamp: $TIMESTAMP"
echo " Safety backup: ./backups/pre_restore_$TIMESTAMP/"
echo ""
echo "⚠️ If you encounter any issues, you can restore the safety backup:"
echo " If you encounter any issues, you can restore the safety backup:"
echo " docker-compose -f timescaledb-compose.yml down"
echo " docker volume rm market_data_timescale_data market_data_redis_data"
echo " docker volume create market_data_timescale_data"

20
COBY/examples/binance_example.py
View File

@@ -50,22 +50,22 @@ class BinanceExample:
def on_status_changed(self, exchange, status):
"""Handle status changes"""
logger.info(f"🔄 {exchange} status changed to: {status.value}")
logger.info(f" {exchange} status changed to: {status.value}")
async def run_example(self):
"""Run the example"""
try:
logger.info("🚀 Starting Binance connector example")
logger.info(" Starting Binance connector example")
# Connect to Binance
logger.info("🔌 Connecting to Binance...")
connected = await self.connector.connect()
if not connected:
logger.error("❌ Failed to connect to Binance")
logger.error(" Failed to connect to Binance")
return
logger.info("✅ Connected to Binance successfully")
logger.info(" Connected to Binance successfully")
# Get available symbols
logger.info("📋 Getting available symbols...")
@@ -94,12 +94,12 @@ class BinanceExample:
if 'BTCUSDT' in symbols:
await self.connector.subscribe_orderbook('BTCUSDT')
await self.connector.subscribe_trades('BTCUSDT')
logger.info("✅ Subscribed to BTCUSDT order book and trades")
logger.info(" Subscribed to BTCUSDT order book and trades")
# Subscribe to ETH order book
if 'ETHUSDT' in symbols:
await self.connector.subscribe_orderbook('ETHUSDT')
logger.info("✅ Subscribed to ETHUSDT order book")
logger.info(" Subscribed to ETHUSDT order book")
# Let it run for a while
logger.info("⏳ Collecting data for 30 seconds...")
@@ -111,7 +111,7 @@ class BinanceExample:
logger.info(f" 📊 Order books received: {self.orderbook_count}")
logger.info(f" 💰 Trades received: {self.trade_count}")
logger.info(f" 📡 Total messages: {stats['message_count']}")
logger.info(f" ❌ Errors: {stats['error_count']}")
logger.info(f" Errors: {stats['error_count']}")
logger.info(f" 🔗 Active streams: {stats['active_streams']}")
logger.info(f" 📋 Subscriptions: {list(stats['subscriptions'].keys())}")
@@ -126,12 +126,12 @@ class BinanceExample:
await self.connector.unsubscribe_orderbook('ETHUSDT')
await self.connector.disconnect()
logger.info("✅ Disconnected successfully")
logger.info(" Disconnected successfully")
except KeyboardInterrupt:
logger.info("⏹️ Interrupted by user")
except Exception as e:
logger.error(f"❌ Example failed: {e}")
logger.error(f" Example failed: {e}")
finally:
# Ensure cleanup
try:
@@ -164,5 +164,5 @@ if __name__ == "__main__":
except KeyboardInterrupt:
print("\n👋 Example stopped by user")
except Exception as e:
print(f"\n❌ Example failed: {e}")
print(f"\n Example failed: {e}")
sys.exit(1)

10
COBY/examples/multi_exchange_example.py
View File

@@ -78,11 +78,11 @@ class MultiExchangeManager:
# Report results
for i, (name, result) in enumerate(zip(self.connectors.keys(), results)):
if isinstance(result, Exception):
logger.error(f"❌ Failed to connect to {name}: {result}")
logger.error(f" Failed to connect to {name}: {result}")
elif result:
logger.info(f"✅ Connected to {name}")
logger.info(f" Connected to {name}")
else:
logger.warning(f"⚠️ Connection to {name} returned False")
logger.warning(f" Connection to {name} returned False")
async def _connect_exchange(self, name: str, connector) -> bool:
"""Connect to a single exchange."""
@@ -111,7 +111,7 @@ class MultiExchangeManager:
# Small delay between subscriptions
await asyncio.sleep(0.5)
else:
logger.warning(f"⚠️ {name} not connected, skipping {symbol}")
logger.warning(f" {name} not connected, skipping {symbol}")
except Exception as e:
logger.error(f"Error subscribing to {symbol} on {name}: {e}")
@@ -152,7 +152,7 @@ class MultiExchangeManager:
for name, connector in self.connectors.items():
try:
await connector.disconnect()
logger.info(f"✅ Disconnected from {name}")
logger.info(f" Disconnected from {name}")
except Exception as e:
logger.error(f"Error disconnecting from {name}: {e}")

26
COBY/health_check.py
View File

@@ -20,7 +20,7 @@ def check_health():
health_data = response.json()
# Basic health check passed
print(f"✅ API Health Check: PASSED")
print(f" API Health Check: PASSED")
print(f" Status: {health_data.get('status', 'unknown')}")
print(f" Timestamp: {health_data.get('timestamp', 'unknown')}")
@@ -30,30 +30,30 @@ def check_health():
for component, status in components.items():
if status.get('healthy', False):
print(f"✅ {component}: HEALTHY")
print(f" {component}: HEALTHY")
else:
print(f"❌ {component}: UNHEALTHY - {status.get('error', 'unknown error')}")
print(f" {component}: UNHEALTHY - {status.get('error', 'unknown error')}")
all_healthy = False
if all_healthy:
print("\n🎉 Overall Health: HEALTHY")
return 0
else:
print("\n⚠️ Overall Health: DEGRADED")
print("\n Overall Health: DEGRADED")
return 1
else:
print(f"❌ API Health Check: FAILED (HTTP {response.status_code})")
print(f" API Health Check: FAILED (HTTP {response.status_code})")
return 1
except requests.exceptions.ConnectionError:
print("❌ API Health Check: FAILED (Connection refused)")
print(" API Health Check: FAILED (Connection refused)")
return 1
except requests.exceptions.Timeout:
print("❌ API Health Check: FAILED (Timeout)")
print(" API Health Check: FAILED (Timeout)")
return 1
except Exception as e:
print(f"❌ API Health Check: FAILED ({str(e)})")
print(f" API Health Check: FAILED ({str(e)})")
return 1
def check_websocket():
@@ -67,14 +67,14 @@ def check_websocket():
sock.close()
if result == 0:
print("✅ WebSocket Server: ACCESSIBLE")
print(" WebSocket Server: ACCESSIBLE")
return True
else:
print("❌ WebSocket Server: NOT ACCESSIBLE")
print(" WebSocket Server: NOT ACCESSIBLE")
return False
except Exception as e:
print(f"❌ WebSocket Server: ERROR ({str(e)})")
print(f" WebSocket Server: ERROR ({str(e)})")
return False
def main():
@@ -94,10 +94,10 @@ def main():
print("🎉 COBY System: FULLY HEALTHY")
return 0
elif api_healthy:
print("⚠️ COBY System: PARTIALLY HEALTHY (API only)")
print(" COBY System: PARTIALLY HEALTHY (API only)")
return 1
else:
print("❌ COBY System: UNHEALTHY")
print(" COBY System: UNHEALTHY")
return 1
if __name__ == "__main__":

2
COBY/test_fixes.py
View File

@@ -148,7 +148,7 @@ async def main():
print("🎉 All tests passed! COBY system should work correctly.")
return 0
else:
print("❌ Some tests failed. Please check the issues above.")
print(" Some tests failed. Please check the issues above.")
return 1
if __name__ == "__main__":

54
COBY/test_integration.py
View File

@@ -33,9 +33,9 @@ async def test_database_connection():
# Test health check
is_healthy = await manager.health_check()
if is_healthy:
logger.info("✅ Database connection: HEALTHY")
logger.info(" Database connection: HEALTHY")
else:
logger.error("❌ Database connection: UNHEALTHY")
logger.error(" Database connection: UNHEALTHY")
return False
# Test storage stats
@@ -49,7 +49,7 @@ async def test_database_connection():
return True
except Exception as e:
logger.error(f"❌ Database test failed: {e}")
logger.error(f" Database test failed: {e}")
return False
@@ -80,17 +80,17 @@ async def test_data_storage():
# Test storing order book
result = await manager.store_orderbook(test_orderbook)
if result:
logger.info("✅ Order book storage: SUCCESS")
logger.info(" Order book storage: SUCCESS")
else:
logger.error("❌ Order book storage: FAILED")
logger.error(" Order book storage: FAILED")
return False
# Test retrieving order book
retrieved = await manager.get_latest_orderbook("BTCUSDT", "test_exchange")
if retrieved:
logger.info(f"✅ Order book retrieval: SUCCESS (mid_price: {retrieved.mid_price})")
logger.info(f" Order book retrieval: SUCCESS (mid_price: {retrieved.mid_price})")
else:
logger.error("❌ Order book retrieval: FAILED")
logger.error(" Order book retrieval: FAILED")
return False
# Create test trade
@@ -107,16 +107,16 @@ async def test_data_storage():
# Test storing trade
result = await manager.store_trade(test_trade)
if result:
logger.info("✅ Trade storage: SUCCESS")
logger.info(" Trade storage: SUCCESS")
else:
logger.error("❌ Trade storage: FAILED")
logger.error(" Trade storage: FAILED")
return False
await manager.close()
return True
except Exception as e:
logger.error(f"❌ Data storage test failed: {e}")
logger.error(f" Data storage test failed: {e}")
return False
@@ -144,9 +144,9 @@ async def test_batch_operations():
# Test batch storage
result = await manager.batch_store_orderbooks(orderbooks)
if result == 5:
logger.info(f"✅ Batch order book storage: SUCCESS ({result} records)")
logger.info(f" Batch order book storage: SUCCESS ({result} records)")
else:
logger.error(f"❌ Batch order book storage: PARTIAL ({result}/5 records)")
logger.error(f" Batch order book storage: PARTIAL ({result}/5 records)")
return False
# Create batch of trades
@@ -166,16 +166,16 @@ async def test_batch_operations():
# Test batch trade storage
result = await manager.batch_store_trades(trades)
if result == 10:
logger.info(f"✅ Batch trade storage: SUCCESS ({result} records)")
logger.info(f" Batch trade storage: SUCCESS ({result} records)")
else:
logger.error(f"❌ Batch trade storage: PARTIAL ({result}/10 records)")
logger.error(f" Batch trade storage: PARTIAL ({result}/10 records)")
return False
await manager.close()
return True
except Exception as e:
logger.error(f"❌ Batch operations test failed: {e}")
logger.error(f" Batch operations test failed: {e}")
return False
@@ -186,31 +186,31 @@ async def test_configuration():
try:
# Test database configuration
db_url = config.get_database_url()
logger.info(f"✅ Database URL: {db_url.replace(config.database.password, '***')}")
logger.info(f" Database URL: {db_url.replace(config.database.password, '***')}")
# Test Redis configuration
redis_url = config.get_redis_url()
logger.info(f"✅ Redis URL: {redis_url.replace(config.redis.password, '***')}")
logger.info(f" Redis URL: {redis_url.replace(config.redis.password, '***')}")
# Test bucket sizes
btc_bucket = config.get_bucket_size('BTCUSDT')
eth_bucket = config.get_bucket_size('ETHUSDT')
logger.info(f"✅ Bucket sizes: BTC=${btc_bucket}, ETH=${eth_bucket}")
logger.info(f" Bucket sizes: BTC=${btc_bucket}, ETH=${eth_bucket}")
# Test configuration dict
config_dict = config.to_dict()
logger.info(f"✅ Configuration loaded: {len(config_dict)} sections")
logger.info(f" Configuration loaded: {len(config_dict)} sections")
return True
except Exception as e:
logger.error(f"❌ Configuration test failed: {e}")
logger.error(f" Configuration test failed: {e}")
return False
async def run_all_tests():
"""Run all integration tests"""
logger.info("🚀 Starting COBY Integration Tests")
logger.info(" Starting COBY Integration Tests")
logger.info("=" * 50)
tests = [
@@ -228,11 +228,11 @@ async def run_all_tests():
result = await test_func()
results.append((test_name, result))
if result:
logger.info(f"✅ {test_name}: PASSED")
logger.info(f" {test_name}: PASSED")
else:
logger.error(f"❌ {test_name}: FAILED")
logger.error(f" {test_name}: FAILED")
except Exception as e:
logger.error(f"❌ {test_name}: ERROR - {e}")
logger.error(f" {test_name}: ERROR - {e}")
results.append((test_name, False))
# Summary
@@ -244,7 +244,7 @@ async def run_all_tests():
total = len(results)
for test_name, result in results:
status = "✅ PASSED" if result else "❌ FAILED"
status = " PASSED" if result else " FAILED"
logger.info(f"{test_name:20} {status}")
logger.info(f"\nOverall: {passed}/{total} tests passed")
@@ -253,7 +253,7 @@ async def run_all_tests():
logger.info("🎉 All tests passed! System is ready.")
return True
else:
logger.error("⚠️ Some tests failed. Check configuration and database connection.")
logger.error(" Some tests failed. Check configuration and database connection.")
return False
@@ -269,6 +269,6 @@ if __name__ == "__main__":
print("The system is ready for the next development phase.")
sys.exit(0)
else:
print("\n❌ Integration tests failed!")
print("\n Integration tests failed!")
print("Please check the logs and fix any issues before proceeding.")
sys.exit(1)

6
COBY/tests/test_all_connectors.py
View File

@@ -226,10 +226,10 @@ async def test_connector_compatibility():
status = connector.get_connection_status()
print(f" ✓ Connection status: {status.value}")
print(f" ✅ {name.upper()} connector passed all tests")
print(f" {name.upper()} connector passed all tests")
except Exception as e:
print(f" ❌ {name.upper()} connector failed: {e}")
print(f" {name.upper()} connector failed: {e}")
print("\n=== All Connector Tests Completed ===")
return True
@@ -278,6 +278,6 @@ if __name__ == "__main__":
async def run_all_tests():
await test_connector_compatibility()
await test_multi_connector_data_flow()
print("✅ All connector tests completed successfully")
print(" All connector tests completed successfully")
asyncio.run(run_all_tests())

8
DASHBOARD_FIXES_SUMMARY.md
View File

@@ -2,7 +2,7 @@
## Issues Fixed
### 1. Empty Chart (No 1m Candlesticks) ✅
### 1. Empty Chart (No 1m Candlesticks)
**Problem**: The 1m candlestick bars disappeared from the chart.
**Root Cause**: The pivot points calculation in `_add_pivot_points_to_chart` was throwing errors and preventing the chart from rendering.
@@ -14,7 +14,7 @@
**Location**: `web/clean_dashboard.py` lines 4365-4369
### 2. Missing Variable Error: `portfolio_str` ✅
### 2. Missing Variable Error: `portfolio_str`
**Problem**: `name 'portfolio_str' is not defined` error in `update_metrics` callback.
**Root Cause**: When adding the Open Interest output, the code that calculated `portfolio_str` and `multiplier_str` was accidentally removed.
@@ -38,7 +38,7 @@ mexc_status = "Connected" if self._check_exchange_connection() else "Disconnecte
**Location**: `web/clean_dashboard.py` lines 1808-1820
### 3. Missing Variable Error: `eth_components` ✅
### 3. Missing Variable Error: `eth_components`
**Problem**: `name 'eth_components' is not defined` error in `update_cob_data` callback.
**Root Cause**: The COB components building code was missing from the callback.
@@ -54,7 +54,7 @@ btc_components = html.Div("BTC COB: Loading...", className="text-muted")
**Note**: This is a temporary fix. The full COB ladder rendering should be restored later.
### 4. Over-Scanning Data Provider ✅
### 4. Over-Scanning Data Provider
**Problem**: Excessive API calls to fetch Open Interest data every 2 seconds (matching the metrics update interval).
**Example from logs**:

6
DATA_STREAM_GUIDE.md
View File

@@ -25,9 +25,9 @@ python check_stream.py snapshot
## What You'll See
### Stream Status Output
- ✅ Dashboard is running
- Dashboard is running
- 📊 Health status
- 🔄 Stream connection and streaming status
- Stream connection and streaming status
- 📈 Total samples and active streams
- 🟢/🔴 Buffer sizes for each data type
@@ -46,7 +46,7 @@ python check_stream.py snapshot
- Bid/ask volumes for each bucket
### Snapshot Output
- ✅ Snapshot saved with filepath
- Snapshot saved with filepath
- 📅 Timestamp of creation
## API Endpoints

84
MODEL_MANAGER_MIGRATION.md
View File

@@ -6,31 +6,31 @@ All model management functionality has been consolidated into a single, unified
## What Was Consolidated
### Files Removed/Migrated:
1. ✅ `utils/model_registry.py` → **CONSOLIDATED**
2. ✅ `utils/checkpoint_manager.py` → **CONSOLIDATED**
3. ✅ `improved_model_saver.py` → **CONSOLIDATED**
4. ✅ `model_checkpoint_saver.py` → **CONSOLIDATED**
5. ✅ `models.py` (legacy registry) → **CONSOLIDATED**
1. `utils/model_registry.py` → **CONSOLIDATED**
2. `utils/checkpoint_manager.py` → **CONSOLIDATED**
3. `improved_model_saver.py` → **CONSOLIDATED**
4. `model_checkpoint_saver.py` → **CONSOLIDATED**
5. `models.py` (legacy registry) → **CONSOLIDATED**
### Classes Consolidated:
1. ✅ `ModelRegistry` (utils/model_registry.py)
2. ✅ `CheckpointManager` (utils/checkpoint_manager.py)
3. ✅ `CheckpointMetadata` (utils/checkpoint_manager.py)
4. ✅ `ImprovedModelSaver` (improved_model_saver.py)
5. ✅ `ModelCheckpointSaver` (model_checkpoint_saver.py)
6. ✅ `ModelRegistry` (models.py - legacy)
1. `ModelRegistry` (utils/model_registry.py)
2. `CheckpointManager` (utils/checkpoint_manager.py)
3. `CheckpointMetadata` (utils/checkpoint_manager.py)
4. `ImprovedModelSaver` (improved_model_saver.py)
5. `ModelCheckpointSaver` (model_checkpoint_saver.py)
6. `ModelRegistry` (models.py - legacy)
## New Unified System
### Primary Class: `ModelManager` (`NN/training/model_manager.py`)
#### Key Features:
- ✅ **Unified Directory Structure**: Uses `@checkpoints/` structure
- ✅ **All Model Types**: CNN, DQN, RL, Transformer, Hybrid
- ✅ **Enhanced Metrics**: Comprehensive performance tracking
- ✅ **Robust Saving**: Multiple fallback strategies
- ✅ **Checkpoint Management**: W&B integration support
- ✅ **Legacy Compatibility**: Maintains all existing APIs
- **Unified Directory Structure**: Uses `@checkpoints/` structure
- **All Model Types**: CNN, DQN, RL, Transformer, Hybrid
- **Enhanced Metrics**: Comprehensive performance tracking
- **Robust Saving**: Multiple fallback strategies
- **Checkpoint Management**: W&B integration support
- **Legacy Compatibility**: Maintains all existing APIs
#### Directory Structure:
```
@@ -72,7 +72,7 @@ from NN.training.model_manager import (
## API Compatibility
### ✅ **Fully Backward Compatible**
### **Fully Backward Compatible**
All existing function calls continue to work:
```python
@@ -83,7 +83,7 @@ save_checkpoint(model, "my_model", "cnn", metrics)
checkpoint = load_best_checkpoint("my_model")
```
### ✅ **Enhanced Functionality**
### **Enhanced Functionality**
New features available through unified interface:
```python
@@ -105,7 +105,7 @@ leaderboard = manager.get_model_leaderboard()
## Files Updated
### ✅ **Core Files Updated:**
### **Core Files Updated:**
1. `core/orchestrator.py` - Uses new ModelManager
2. `web/clean_dashboard.py` - Updated imports
3. `NN/models/dqn_agent.py` - Updated imports
@@ -113,7 +113,7 @@ leaderboard = manager.get_model_leaderboard()
5. `tests/test_training.py` - Updated imports
6. `main.py` - Updated imports
### ✅ **Backup Created:**
### **Backup Created:**
All old files moved to `backup/old_model_managers/` for reference.
## Benefits Achieved
@@ -124,37 +124,37 @@ All old files moved to `backup/old_model_managers/` for reference.
- **Reduction**: ~60% code duplication eliminated
### 🔧 **Maintenance:**
- ✅ Single source of truth for model management
- ✅ Consistent API across all model types
- ✅ Centralized configuration and settings
- ✅ Unified error handling and logging
- Single source of truth for model management
- Consistent API across all model types
- Centralized configuration and settings
- Unified error handling and logging
### 🚀 **Enhanced Features:**
- ✅ `@checkpoints/` directory structure
- ✅ W&B integration support
- ✅ Enhanced performance metrics
- ✅ Multiple save strategies with fallbacks
- ✅ Comprehensive checkpoint management
### **Enhanced Features:**
- `@checkpoints/` directory structure
- W&B integration support
- Enhanced performance metrics
- Multiple save strategies with fallbacks
- Comprehensive checkpoint management
### 🔄 **Compatibility:**
- ✅ Zero breaking changes for existing code
- ✅ All existing APIs preserved
- ✅ Legacy function calls still work
- ✅ Gradual migration path available
### **Compatibility:**
- Zero breaking changes for existing code
- All existing APIs preserved
- Legacy function calls still work
- Gradual migration path available
## Migration Verification
### ✅ **Test Commands:**
### **Test Commands:**
```bash
# Test the new unified system
cd /mnt/shared/DEV/repos/d-popov.com/gogo2
python -c "from NN.training.model_manager import create_model_manager; m = create_model_manager(); print('✅ ModelManager works')"
python -c "from NN.training.model_manager import create_model_manager; m = create_model_manager(); print(' ModelManager works')"
# Test legacy compatibility
python -c "from NN.training.model_manager import save_model, load_model; print('✅ Legacy functions work')"
python -c "from NN.training.model_manager import save_model, load_model; print(' Legacy functions work')"
```
### ✅ **Integration Tests:**
### **Integration Tests:**
- Clean dashboard loads without errors
- Model saving/loading works correctly
- Checkpoint management functions properly
@@ -176,8 +176,8 @@ If any issues arise, the old files are preserved in `backup/old_model_managers/`
---
**Status**: ✅ **MIGRATION COMPLETE**
**Status**: **MIGRATION COMPLETE**
**Date**: $(date)
**Files Consolidated**: 5 → 1
**Code Reduction**: ~60%
**Compatibility**: ✅ 100% Backward Compatible
**Compatibility**: 100% Backward Compatible

2
NN/training/enhanced_rl_training_integration.py
View File

@@ -363,7 +363,7 @@ class EnhancedRLTrainingIntegrator:
# Integration status
if self.training_stats['comprehensive_features_used'] > 0:
logger.info("STATUS: COMPREHENSIVE RL TRAINING INTEGRATION SUCCESSFUL! ✅")
logger.info("STATUS: COMPREHENSIVE RL TRAINING INTEGRATION SUCCESSFUL! ")
logger.info("The system is now using the full 13,400 feature comprehensive state.")
else:
logger.warning("STATUS: Integration partially successful - some fallbacks may occur")

34
PREDICTION_LOOP_INTEGRATION.md
View File

@@ -23,13 +23,13 @@
2. **Calls ALL Models**
```python
# ✅ CNN Model
# CNN Model
cnn_prediction = self.orchestrator.cnn_model.predict(cnn_features)
# ✅ DQN Agent
# DQN Agent
action = self.orchestrator.dqn_agent.act(dqn_state)
# ✅ COB RL
# COB RL
cob_prediction = self.orchestrator.cob_rl_model.predict(cob_features)
```
@@ -86,9 +86,9 @@ class TradingOrchestrator:
)
if result:
logger.info(f"✅ Signal executed successfully")
logger.info(f" Signal executed successfully")
else:
logger.warning(f"⚠️ Signal execution failed or blocked")
logger.warning(f" Signal execution failed or blocked")
except Exception as e:
logger.error(f"Error processing prediction signal: {e}")
@@ -116,7 +116,7 @@ class CleanTradingDashboard:
name="PredictionLoop"
)
self.prediction_thread.start()
logger.info("🔄 Real-time prediction loop started in background")
logger.info(" Real-time prediction loop started in background")
```
### Step 3: Update Data Provider (if needed)
@@ -151,22 +151,22 @@ async def get_latest_candles(self, symbol: str, timeframe: str, limit: int = 100
INFO:web.clean_dashboard:🔗 Running initial chained inference...
INFO:web.clean_dashboard:🔗 Running chained inference for ETH/USDT with 10 steps
WARNING:core.orchestrator:No model predictions available for step 0
WARNING:web.clean_dashboard:❌ Chained inference returned no predictions
WARNING:web.clean_dashboard: Chained inference returned no predictions
# Then nothing... models never called again
```
### After (Expected):
```
INFO:core.realtime_prediction_loop:🔄 Starting Real-Time Prediction Loop
INFO:core.realtime_prediction_loop:🔄 Prediction loop started for ETH/USDT
INFO:core.realtime_prediction_loop:🔄 Prediction loop started for BTC/USDT
INFO:core.realtime_prediction_loop: Starting Real-Time Prediction Loop
INFO:core.realtime_prediction_loop: Prediction loop started for ETH/USDT
INFO:core.realtime_prediction_loop: Prediction loop started for BTC/USDT
# Every 1-5 seconds:
INFO:core.realtime_prediction_loop:📊 New 1s candle detected for ETH/USDT - running predictions
INFO:core.realtime_prediction_loop:✅ CNN prediction for ETH/USDT: {'action': 'BUY', 'confidence': 0.78}
INFO:core.realtime_prediction_loop:✅ DQN prediction for ETH/USDT: BUY
INFO:core.realtime_prediction_loop:✅ COB RL prediction for ETH/USDT: {'action': 'BUY', 'confidence': 0.82}
INFO:core.realtime_prediction_loop: CNN prediction for ETH/USDT: {'action': 'BUY', 'confidence': 0.78}
INFO:core.realtime_prediction_loop: DQN prediction for ETH/USDT: BUY
INFO:core.realtime_prediction_loop: COB RL prediction for ETH/USDT: {'action': 'BUY', 'confidence': 0.82}
INFO:core.realtime_prediction_loop:📤 Trading signal sent for ETH/USDT: BUY (confidence: 0.80, trigger: new_1s_candle)
INFO:core.orchestrator:📥 Received prediction signal: BUY for ETH/USDT (conf: 0.80)
@@ -180,10 +180,10 @@ INFO:core.trading_executor:Executing BUY: 0.050000 ETH/USDT at $4191.25
### Check if Prediction Loop is Running:
```python
# In dashboard or orchestrator logs, look for:
- "🔄 Real-time prediction loop started"
- " Real-time prediction loop started"
- "📊 New 1s candle detected" (every second)
- "✅ CNN prediction" (model being called!)
- "✅ DQN prediction" (model being called!)
- " CNN prediction" (model being called!)
- " DQN prediction" (model being called!)
- "📤 Trading signal sent"
```
@@ -208,7 +208,7 @@ cob_predictions_count = 0
## Files Modified
1. ✅ **core/realtime_prediction_loop.py** (NEW)
1. **core/realtime_prediction_loop.py** (NEW)
- Created prediction loop
2. ⏳ **core/orchestrator.py** (TO MODIFY)

10
TODO.md
View File

@@ -1,4 +1,4 @@
# 🚀 GOGO2 Enhanced Trading System - TODO
# GOGO2 Enhanced Trading System - TODO
## 🎯 **IMMEDIATE PRIORITIES** (System Stability & Core Performance)
@@ -206,26 +206,26 @@ initial dash loads 180 historical candles, but then we drop them when we get the
use existing checkpoint manager if it;s not too bloated as well. otherwise re-implement clean one where we keep rotate up to 5 checkpoints - best if we can reliably measure performance, otherwise latest 5
### **✅ Trading Integration**
### ** Trading Integration**
- [ ] Recent signals show with confidence levels
- [ ] Manual BUY/SELL buttons work
- [ ] Executed vs blocked signals displayed
- [ ] Current position shows correctly
- [ ] Session P&L updates in real-time
### **✅ COB Integration**
### ** COB Integration**
- [ ] System status shows "COB: Active"
- [ ] ETH/USDT COB data displays
- [ ] BTC/USDT COB data displays
- [ ] Order book metrics update
### **✅ Training Pipeline**
### ** Training Pipeline**
- [ ] CNN model status shows "Active"
- [ ] RL model status shows "Training"
- [ ] Training metrics update
- [ ] Model performance data available
### **✅ Performance**
### ** Performance**
- [ ] Chart updates every second
- [ ] No flickering or data loss
- [ ] WebSocket connection stable

12
TRAINING_FIX_IMPLEMENTATION.md
View File

@@ -7,7 +7,7 @@
## Critical Issues Identified
### 1. Division by Zero ✅ FIXED
### 1. Division by Zero FIXED
**Problem**: Trading executor crashed when price was 0 or invalid
**Solution**: Added price validation before division in `core/trading_executor.py`
```python
@@ -16,14 +16,14 @@ if current_price <= 0:
return False
```
### 2. Mock Predictions ✅ FIXED
### 2. Mock Predictions FIXED
**Problem**: System fell back to "mock predictions" when training unavailable (POLICY VIOLATION!)
**Solution**: Removed mock fallback, system now fails gracefully
```python
logger.error("CRITICAL: Enhanced training system not available - predictions disabled. NEVER use mock data.")
```
### 3. Torch Import ✅ ALREADY FIXED
### 3. Torch Import ALREADY FIXED
**Problem**: "cannot access local variable 'torch'" error
**Status**: Already has None placeholder when import fails
@@ -182,14 +182,14 @@ def train_on_trade_outcome(self, signal_data, trade_record, reward):
self._train_cob_on_outcome(cob_data, action, reward, weight)
logger.info(f"COB RL trained with weight {weight:.2f}")
logger.info(f"✅ TRAINED ALL MODELS: PnL=${trade_record.pnl:.2f}, Reward={reward:.4f}")
logger.info(f" TRAINED ALL MODELS: PnL=${trade_record.pnl:.2f}, Reward={reward:.4f}")
```
---
## Implementation Steps
### Phase 1: Core Infrastructure (Priority 1) ✅
### Phase 1: Core Infrastructure (Priority 1)
- [x] Fix division by zero
- [x] Remove mock predictions
- [x] Fix torch imports
@@ -256,7 +256,7 @@ def train_on_trade_outcome(self, signal_data, trade_record, reward):
### Log Messages to Watch:
```
✅ TRAINED ALL MODELS: PnL=$2.35, Reward=25.40
TRAINED ALL MODELS: PnL=$2.35, Reward=25.40
REWARD CALC: PnL=0.0235, Time=-0.002, Risk=1.15, Final=25.40
CNN trained with weight 0.35
DQN trained with weight 0.45, loss=0.0123

8
_dev/dev_notes.md
View File

@@ -48,26 +48,26 @@ initial dash loads 180 historical candles, but then we drop them when we get the
use existing checkpoint manager if it;s not too bloated as well. otherwise re-implement clean one where we keep rotate up to 5 checkpoints - best if we can reliably measure performance, otherwise latest 5
### **✅ Trading Integration**
### ** Trading Integration**
- [ ] Recent signals show with confidence levels
- [ ] Manual BUY/SELL buttons work
- [ ] Executed vs blocked signals displayed
- [ ] Current position shows correctly
- [ ] Session P&L updates in real-time
### **✅ COB Integration**
### ** COB Integration**
- [ ] System status shows "COB: Active"
- [ ] ETH/USDT COB data displays
- [ ] BTC/USDT COB data displays
- [ ] Order book metrics update
### **✅ Training Pipeline**
### ** Training Pipeline**
- [ ] CNN model status shows "Active"
- [ ] RL model status shows "Training"
- [ ] Training metrics update
- [ ] Model performance data available
### **✅ Performance**
### ** Performance**
- [ ] Chart updates every second
- [ ] No flickering or data loss
- [ ] WebSocket connection stable

44
core/data_provider.py
View File

@@ -277,7 +277,7 @@ class DataProvider:
if DUCKDB_STORAGE_AVAILABLE:
try:
self.duckdb_storage = DuckDBStorage()
logger.info("✅ DuckDB storage initialized (unified Parquet + SQL)")
logger.info(" DuckDB storage initialized (unified Parquet + SQL)")
except Exception as e:
logger.warning(f"Could not initialize DuckDB storage: {e}")
@@ -396,7 +396,7 @@ class DataProvider:
if success:
self._unified_storage_enabled = True
logger.info("✅ Unified storage system enabled successfully")
logger.info(" Unified storage system enabled successfully")
return True
else:
logger.error("Failed to enable unified storage system")
@@ -550,7 +550,7 @@ class DataProvider:
else:
logger.info("Skipping initial data load (using DuckDB cache)")
logger.info("✅ Initial data load completed - stopping maintenance worker")
logger.info(" Initial data load completed - stopping maintenance worker")
logger.info("📊 Data will be updated on-demand only (no continuous fetching)")
# Stop the maintenance worker after initial load
@@ -602,7 +602,7 @@ class DataProvider:
# Cap at 1500 candles maximum
fetch_limit = min(estimated_missing + 10, 1500)
logger.info(f"🔄 Fetching {fetch_limit} recent candles for {symbol} {timeframe} (since {last_timestamp})")
logger.info(f" Fetching {fetch_limit} recent candles for {symbol} {timeframe} (since {last_timestamp})")
new_df = self._fetch_from_binance(symbol, timeframe, fetch_limit)
if new_df is None or new_df.empty:
@@ -622,9 +622,9 @@ class DataProvider:
combined_df = combined_df.sort_index()
self.cached_data[symbol][timeframe] = combined_df.tail(1500)
logger.info(f"✅ {symbol} {timeframe}: +{len(new_df)} new (total: {len(self.cached_data[symbol][timeframe])})")
logger.info(f" {symbol} {timeframe}: +{len(new_df)} new (total: {len(self.cached_data[symbol][timeframe])})")
else:
logger.info(f"✅ {symbol} {timeframe}: Up to date ({len(existing_df)} candles)")
logger.info(f" {symbol} {timeframe}: Up to date ({len(existing_df)} candles)")
else:
# No existing data - fetch initial 1500 candles
logger.info(f"🆕 No existing data, fetching 1500 candles for {symbol} {timeframe}")
@@ -643,7 +643,7 @@ class DataProvider:
with self.data_lock:
self.cached_data[symbol][timeframe] = df
logger.info(f"✅ Loaded {len(df)} candles for {symbol} {timeframe}")
logger.info(f" Loaded {len(df)} candles for {symbol} {timeframe}")
# Small delay to avoid rate limits
time.sleep(0.1)
@@ -651,7 +651,7 @@ class DataProvider:
except Exception as e:
logger.error(f"Error loading data for {symbol} {timeframe}: {e}")
logger.info("✅ Smart incremental data load completed")
logger.info(" Smart incremental data load completed")
def _start_background_catch_up(self):
"""
@@ -737,9 +737,9 @@ class DataProvider:
final_count = len(self.cached_data[symbol][timeframe])
logger.info(f"✅ {symbol} {timeframe}: Caught up! Now have {final_count} candles")
logger.info(f" {symbol} {timeframe}: Caught up! Now have {final_count} candles")
else:
logger.warning(f"❌ {symbol} {timeframe}: Could not fetch historical data from any exchange")
logger.warning(f" {symbol} {timeframe}: Could not fetch historical data from any exchange")
except Exception as e:
logger.error(f"Error catching up candles for {symbol} {timeframe}: {e}")
@@ -775,7 +775,7 @@ class DataProvider:
# Cap at 1500 candles maximum
fetch_limit = min(estimated_missing + 5, 1500)
logger.info(f"🔄 Fetching {fetch_limit} recent candles for {symbol} {timeframe} (since {last_timestamp})")
logger.info(f" Fetching {fetch_limit} recent candles for {symbol} {timeframe} (since {last_timestamp})")
# Fetch missing candles
df = self._fetch_from_binance(symbol, timeframe, fetch_limit)
@@ -811,7 +811,7 @@ class DataProvider:
candle_count = len(self.cached_data[symbol][timeframe])
logger.info(f"✅ Updated {symbol} {timeframe}: +{len(df)} new (total: {candle_count})")
logger.info(f" Updated {symbol} {timeframe}: +{len(df)} new (total: {candle_count})")
else:
logger.warning(f"Could not fetch new data for {symbol} {timeframe}")
else:
@@ -827,17 +827,17 @@ class DataProvider:
try:
if symbol and timeframe:
# Refresh specific symbol/timeframe
logger.info(f"🔄 Manual refresh requested for {symbol} {timeframe}")
logger.info(f" Manual refresh requested for {symbol} {timeframe}")
self._update_cached_data(symbol, timeframe)
else:
# Refresh all symbols/timeframes
logger.info("🔄 Manual refresh requested for all symbols/timeframes")
logger.info(" Manual refresh requested for all symbols/timeframes")
for sym in self.symbols:
for tf in self.timeframes:
self._update_cached_data(sym, tf)
time.sleep(0.1) # Small delay to avoid rate limits
logger.info("✅ Manual refresh completed for all symbols/timeframes")
logger.info(" Manual refresh completed for all symbols/timeframes")
except Exception as e:
logger.error(f"Error in manual refresh: {e}")
@@ -3107,7 +3107,7 @@ class DataProvider:
def _load_from_duckdb_sync(self):
"""Load all data from DuckDB synchronously for instant startup"""
if not self.duckdb_storage:
logger.warning("⚠️ DuckDB storage not available - cannot load cached data")
logger.warning(" DuckDB storage not available - cannot load cached data")
return
logger.info("📦 Loading cached data from DuckDB...")
@@ -3125,18 +3125,18 @@ class DataProvider:
if df is not None and not df.empty:
with self.data_lock:
self.cached_data[symbol][timeframe] = df.tail(1500)
logger.info(f"✅ {symbol} {timeframe}: {len(df)} candles from DuckDB")
logger.info(f" {symbol} {timeframe}: {len(df)} candles from DuckDB")
loaded_count += len(df)
else:
logger.debug(f"No data in DuckDB for {symbol} {timeframe} - will fetch from API")
except Exception as e:
logger.error(f"❌ Error loading {symbol} {timeframe}: {e}")
logger.error(f" Error loading {symbol} {timeframe}: {e}")
if loaded_count > 0:
logger.info(f"✅ Loaded {loaded_count:,} candles total")
logger.info(f" Loaded {loaded_count:,} candles total")
else:
logger.warning("⚠️ No cached data found - will fetch from API")
logger.warning(" No cached data found - will fetch from API")
def _load_from_duckdb(self, symbol: str, timeframe: str, limit: int = 1500) -> Optional[pd.DataFrame]:
"""Load data from DuckDB storage
@@ -3338,7 +3338,7 @@ class DataProvider:
async def _start_fallback_websocket_streaming(self):
"""Fallback to old WebSocket method if Enhanced COB WebSocket fails"""
try:
logger.warning("⚠️ Starting fallback WebSocket streaming")
logger.warning(" Starting fallback WebSocket streaming")
# Start old WebSocket for each symbol
for symbol in self.symbols:
@@ -3346,7 +3346,7 @@ class DataProvider:
self.websocket_tasks[symbol] = task
except Exception as e:
logger.error(f"❌ Error starting fallback WebSocket: {e}")
logger.error(f" Error starting fallback WebSocket: {e}")
def get_cob_websocket_status(self) -> Dict[str, Any]:
"""Get COB WebSocket status for dashboard"""

2
core/enhanced_cob_websocket.py
View File

@@ -364,7 +364,7 @@ class EnhancedCOBWebSocket:
try:
await callback(symbol, cob_data)
except Exception as e:
logger.error(f"❌ Error in COB callback: {e}")
logger.error(f" Error in COB callback: {e}")
logger.debug(f"Initial snapshot for {symbol}: ${mid_price:.2f}, spread: {spread_bps:.1f} bps")
else:

8
core/exchanges/bybit_rest_client.py
View File

@@ -324,10 +324,10 @@ class BybitRestClient:
"""
try:
result = self.get_server_time()
logger.info("✅ Bybit REST API connectivity test successful")
logger.info(" Bybit REST API connectivity test successful")
return True
except Exception as e:
logger.error(f"❌ Bybit REST API connectivity test failed: {e}")
logger.error(f" Bybit REST API connectivity test failed: {e}")
return False
def test_authentication(self) -> bool:
@@ -338,8 +338,8 @@ class BybitRestClient:
"""
try:
result = self.get_account_info()
logger.info("✅ Bybit REST API authentication test successful")
logger.info(" Bybit REST API authentication test successful")
return True
except Exception as e:
logger.error(f"❌ Bybit REST API authentication test failed: {e}")
logger.error(f" Bybit REST API authentication test failed: {e}")
return False

2
core/exchanges/exchanges_research_report.md
View File

@@ -256,7 +256,7 @@ Each exchange was scored (1-5) across four weighted categories:
| Criteria | Deribit | Binance | Bybit | OKX |
|-------------------|---------------|---------------|---------------|---------------|
| **Max Leverage** | 10× | 75× | 100× | 10× |
| **Market Orders** | ✅ | ✅ | ✅ | ✅ |
| **Market Orders** | | | | |
| **Base Fee** | 0% maker | 0.02% maker | -0.01% maker | 0.02% maker |
| **Python SDK** | Official | Robust | Low-latency | Full-featured |
| **HFT Suitability**| ★★★★☆ | ★★★★★ | ★★★★☆ | ★★★☆☆ |

6
core/exchanges/mexc/debug/final_mexc_order_test.py
View File

@@ -27,7 +27,7 @@ def test_final_mexc_order():
api_secret = os.getenv('MEXC_SECRET_KEY', '')
if not api_key or not api_secret:
print("❌ No MEXC API credentials found")
print(" No MEXC API credentials found")
return
# Parameters
@@ -68,9 +68,9 @@ def test_final_mexc_order():
print(f"Response: {response.text}")
if response.status_code == 200:
print("✅ SUCCESS!")
print(" SUCCESS!")
else:
print("❌ FAILED")
print(" FAILED")
# Try alternative method - sending as query params
print("\n--- Trying alternative method ---")
test_alternative_method(api_key, api_secret)

8
core/exchanges/mexc/debug/fix_mexc_orders.py
View File

@@ -59,7 +59,7 @@ def test_mexc_order_placement():
api_secret = os.getenv('MEXC_SECRET_KEY', '')
if not api_key or not api_secret:
print("❌ No MEXC API credentials found")
print(" No MEXC API credentials found")
return
# Test parameters - very small order
@@ -105,7 +105,7 @@ def test_mexc_order_placement():
if response.status_code == 200:
result = response.json()
print("✅ Order placed successfully!")
print(" Order placed successfully!")
print(f"Order result: {result}")
# Try to cancel it immediately if we got an order ID
@@ -131,11 +131,11 @@ def test_mexc_order_placement():
print(f"Cancel response: {cancel_response.status_code} - {cancel_response.text}")
else:
print("❌ Order placement failed")
print(" Order placement failed")
print(f"Response: {response.text}")
except Exception as e:
print(f"❌ Request error: {e}")
print(f" Request error: {e}")
if __name__ == "__main__":
test_mexc_order_placement()

10
core/exchanges/mexc/debug/fix_mexc_orders_v2.py
View File

@@ -59,7 +59,7 @@ def test_mexc_order_v2():
api_secret = os.getenv('MEXC_SECRET_KEY', '')
if not api_key or not api_secret:
print("❌ No MEXC API credentials found")
print(" No MEXC API credentials found")
return
# Order parameters matching MEXC examples
@@ -99,19 +99,19 @@ def test_mexc_order_v2():
if response.status_code == 200:
result = response.json()
print("✅ Order placed successfully!")
print(" Order placed successfully!")
print(f"Order result: {result}")
# Cancel immediately if successful
if 'orderId' in result:
print(f"\n🔄 Canceling order {result['orderId']}...")
print(f"\n Canceling order {result['orderId']}...")
cancel_order(api_key, api_secret, 'ETHUSDC', result['orderId'])
else:
print("❌ Order placement failed")
print(" Order placement failed")
except Exception as e:
print(f"❌ Request error: {e}")
print(f" Request error: {e}")
def cancel_order(api_key: str, secret_key: str, symbol: str, order_id: str):
"""Cancel a MEXC order"""

10
core/exchanges/mexc/debug/fix_mexc_orders_v3.py
View File

@@ -42,7 +42,7 @@ def test_mexc_order_v3():
api_secret = os.getenv('MEXC_SECRET_KEY', '')
if not api_key or not api_secret:
print("❌ No MEXC API credentials found")
print(" No MEXC API credentials found")
return
# Order parameters exactly like the examples
@@ -92,19 +92,19 @@ def test_mexc_order_v3():
if response.status_code == 200:
result = response.json()
print("✅ Order placed successfully!")
print(" Order placed successfully!")
print(f"Order result: {result}")
# Cancel immediately if successful
if 'orderId' in result:
print(f"\n🔄 Canceling order {result['orderId']}...")
print(f"\n Canceling order {result['orderId']}...")
cancel_order_v3(api_key, api_secret, 'ETHUSDC', result['orderId'])
else:
print("❌ Order placement failed")
print(" Order placement failed")
except Exception as e:
print(f"❌ Request error: {e}")
print(f" Request error: {e}")
def cancel_order_v3(api_key: str, secret_key: str, symbol: str, order_id: str):
"""Cancel a MEXC order using V3 method"""

22
core/exchanges/mexc/debug/test_mexc_interface_debug.py
View File

@@ -26,7 +26,7 @@ def debug_interface():
api_secret = os.getenv('MEXC_SECRET_KEY', '')
if not api_key or not api_secret:
print("❌ No MEXC API credentials found")
print(" No MEXC API credentials found")
return False
from NN.exchanges.mexc_interface import MEXCInterface
@@ -98,17 +98,17 @@ def debug_interface():
# Compare parameters
print(f"\n📊 COMPARISON:")
print(f"symbol: Interface='{interface_params['symbol']}', Manual='{manual_params['symbol']}' {'✅' if interface_params['symbol'] == manual_params['symbol'] else '❌'}")
print(f"side: Interface='{interface_params['side']}', Manual='{manual_params['side']}' {'✅' if interface_params['side'] == manual_params['side'] else '❌'}")
print(f"type: Interface='{interface_params['type']}', Manual='{manual_params['type']}' {'✅' if interface_params['type'] == manual_params['type'] else '❌'}")
print(f"quantity: Interface='{interface_params['quantity']}', Manual='{manual_params['quantity']}' {'✅' if interface_params['quantity'] == manual_params['quantity'] else '❌'}")
print(f"price: Interface='{interface_params['price']}', Manual='{manual_params['price']}' {'✅' if interface_params['price'] == manual_params['price'] else '❌'}")
print(f"timestamp: Interface='{interface_params['timestamp']}', Manual='{manual_params['timestamp']}' {'✅' if interface_params['timestamp'] == manual_params['timestamp'] else '❌'}")
print(f"recvWindow: Interface='{interface_params['recvWindow']}', Manual='{manual_params['recvWindow']}' {'✅' if interface_params['recvWindow'] == manual_params['recvWindow'] else '❌'}")
print(f"symbol: Interface='{interface_params['symbol']}', Manual='{manual_params['symbol']}' {'' if interface_params['symbol'] == manual_params['symbol'] else ''}")
print(f"side: Interface='{interface_params['side']}', Manual='{manual_params['side']}' {'' if interface_params['side'] == manual_params['side'] else ''}")
print(f"type: Interface='{interface_params['type']}', Manual='{manual_params['type']}' {'' if interface_params['type'] == manual_params['type'] else ''}")
print(f"quantity: Interface='{interface_params['quantity']}', Manual='{manual_params['quantity']}' {'' if interface_params['quantity'] == manual_params['quantity'] else ''}")
print(f"price: Interface='{interface_params['price']}', Manual='{manual_params['price']}' {'' if interface_params['price'] == manual_params['price'] else ''}")
print(f"timestamp: Interface='{interface_params['timestamp']}', Manual='{manual_params['timestamp']}' {'' if interface_params['timestamp'] == manual_params['timestamp'] else ''}")
print(f"recvWindow: Interface='{interface_params['recvWindow']}', Manual='{manual_params['recvWindow']}' {'' if interface_params['recvWindow'] == manual_params['recvWindow'] else ''}")
# Check for timeInForce difference
if 'timeInForce' in interface_params:
print(f"timeInForce: Interface='{interface_params['timeInForce']}', Manual=None ❌ (EXTRA PARAMETER)")
print(f"timeInForce: Interface='{interface_params['timeInForce']}', Manual=None (EXTRA PARAMETER)")
# Test without timeInForce
print(f"\n🔧 TESTING WITHOUT timeInForce:")
@@ -119,10 +119,10 @@ def debug_interface():
print(f"Interface signature (no timeInForce): {interface_signature_minimal}")
if interface_signature_minimal == manual_signature:
print("✅ Signatures match when timeInForce is removed!")
print(" Signatures match when timeInForce is removed!")
return True
else:
print("❌ Still don't match")
print(" Still don't match")
return False

12
core/exchanges/mexc/debug/test_mexc_order_signature.py
View File

@@ -31,7 +31,7 @@ def test_order_signature():
api_secret = os.getenv('MEXC_SECRET_KEY', '')
if not api_key or not api_secret:
print("❌ No MEXC API credentials found")
print(" No MEXC API credentials found")
return False
# Test order parameters
@@ -79,9 +79,9 @@ def test_order_signature():
# Compare
if signature_manual == signature_interface:
print("✅ Signatures match!")
print(" Signatures match!")
else:
print("❌ Signatures don't match")
print(" Signatures don't match")
print("This indicates a problem with the signature generation method")
return False
@@ -106,10 +106,10 @@ def test_order_signature():
print(f"Response: {response.text}")
if response.status_code == 200:
print("✅ Manual order method works!")
print(" Manual order method works!")
return True
else:
print("❌ Manual order method failed")
print(" Manual order method failed")
# Test 4: Try test order endpoint
print("\n4. Testing with test order endpoint:")
@@ -119,7 +119,7 @@ def test_order_signature():
print(f"Test order response: {response2.status_code} - {response2.text}")
if response2.status_code == 200:
print("✅ Test order works - real order parameters might have issues")
print(" Test order works - real order parameters might have issues")
# Test 5: Try different parameter variations
print("\n5. Testing different parameter sets:")

12
core/exchanges/mexc/debug/test_mexc_order_signature_v2.py
View File

@@ -28,7 +28,7 @@ def test_different_approaches():
api_secret = os.getenv('MEXC_SECRET_KEY', '')
if not api_key or not api_secret:
print("❌ No MEXC API credentials found")
print(" No MEXC API credentials found")
return False
# Test order parameters
@@ -109,13 +109,13 @@ def test_different_approaches():
print(f"Response: {response.status_code} - {response.text}")
if response.status_code == 200:
print(f"✅ {method} WORKS!")
print(f" {method} WORKS!")
return True
else:
print(f"❌ {method} failed")
print(f" {method} failed")
except Exception as e:
print(f"❌ {method} error: {e}")
print(f" {method} error: {e}")
# Try one more approach - use minimal parameters
print("\n" + "=" * 60)
@@ -149,11 +149,11 @@ def test_different_approaches():
print(f"Minimal response: {response.status_code} - {response.text}")
if response.status_code == 200:
print("✅ Minimal parameters work!")
print(" Minimal parameters work!")
return True
except Exception as e:
print(f"❌ Minimal parameters error: {e}")
print(f" Minimal parameters error: {e}")
return False

10
core/exchanges/mexc/debug/test_mexc_signature_debug.py
View File

@@ -30,7 +30,7 @@ def test_signature_generation():
api_secret = os.getenv('MEXC_SECRET_KEY', '')
if not api_key or not api_secret:
print("❌ No MEXC API credentials found")
print(" No MEXC API credentials found")
return False
# Import the interface
@@ -66,9 +66,9 @@ def test_signature_generation():
# Compare
if signature_manual == signature_interface:
print("✅ Signatures match!")
print(" Signatures match!")
else:
print("❌ Signatures don't match")
print(" Signatures don't match")
print("This indicates a problem with the signature generation method")
# Test 3: Try account request with manual signature
@@ -97,10 +97,10 @@ def test_signature_generation():
print(f"Response: {response.text}")
if response.status_code == 200:
print("✅ Manual method works!")
print(" Manual method works!")
return True
else:
print("❌ Manual method failed")
print(" Manual method failed")
# Test 4: Try different parameter ordering
print("\n4. Testing different parameter orderings:")

14
core/exchanges/mexc/debug/test_small_mexc_order.py
View File

@@ -24,22 +24,22 @@ def test_small_order():
api_secret = os.getenv('MEXC_SECRET_KEY', '')
if not api_key or not api_secret:
print("❌ No MEXC API credentials found")
print(" No MEXC API credentials found")
return
# Create MEXC interface
mexc = MEXCInterface(api_key=api_key, api_secret=api_secret, test_mode=False)
if not mexc.connect():
print("❌ Failed to connect to MEXC API")
print(" Failed to connect to MEXC API")
return
print("✅ Connected to MEXC API")
print(" Connected to MEXC API")
# Get current price
ticker = mexc.get_ticker("ETH/USDT") # Will be converted to ETHUSDC
if not ticker:
print("❌ Failed to get ticker")
print(" Failed to get ticker")
return
current_price = ticker['last']
@@ -63,7 +63,7 @@ def test_small_order():
)
if result:
print("✅ Order placed successfully!")
print(" Order placed successfully!")
print(f"Order result: {result}")
# Try to cancel it immediately
@@ -72,10 +72,10 @@ def test_small_order():
cancel_result = mexc.cancel_order("ETH/USDT", result['orderId'])
print(f"Cancel result: {cancel_result}")
else:
print("❌ Order placement failed")
print(" Order placement failed")
except Exception as e:
print(f"❌ Order error: {e}")
print(f" Order error: {e}")
if __name__ == "__main__":
test_small_order()

20
core/exchanges/mexc/test_live_trading.py
View File

@@ -106,9 +106,9 @@ async def test_live_trading():
if user_input.upper() == 'YES':
cancelled = executor._cancel_open_orders("ETH/USDT")
if cancelled:
logger.info("✅ Open orders cancelled successfully")
logger.info(" Open orders cancelled successfully")
else:
logger.warning("⚠️ Some orders may not have been cancelled")
logger.warning(" Some orders may not have been cancelled")
else:
logger.info("No open orders found")
except Exception as e:
@@ -146,7 +146,7 @@ async def test_live_trading():
)
if success:
logger.info("✅ Test BUY order executed successfully!")
logger.info(" Test BUY order executed successfully!")
# Check order status
await asyncio.sleep(1)
@@ -168,14 +168,14 @@ async def test_live_trading():
)
if success:
logger.info("✅ Test SELL order executed successfully!")
logger.info("✅ Full test trade cycle completed!")
logger.info(" Test SELL order executed successfully!")
logger.info(" Full test trade cycle completed!")
else:
logger.warning("❌ Test SELL order failed")
logger.warning(" Test SELL order failed")
else:
logger.warning("❌ No position found after BUY order")
logger.warning(" No position found after BUY order")
else:
logger.warning("❌ Test BUY order failed")
logger.warning(" Test BUY order failed")
except Exception as e:
logger.error(f"Error executing test trade: {e}")
@@ -205,7 +205,7 @@ async def test_live_trading():
try:
open_orders = executor.exchange.get_open_orders("ETH/USDT")
if open_orders and len(open_orders) > 0:
logger.warning(f"⚠️ {len(open_orders)} open orders still pending:")
logger.warning(f" {len(open_orders)} open orders still pending:")
for order in open_orders:
order_id = order.get('orderId', 'N/A')
side = order.get('side', 'N/A')
@@ -214,7 +214,7 @@ async def test_live_trading():
status = order.get('status', 'N/A')
logger.info(f" Order {order_id}: {side} {qty} ETH at ${price} - Status: {status}")
else:
logger.info("✅ No pending orders")
logger.info(" No pending orders")
except Exception as e:
logger.error(f"Error checking final open orders: {e}")

6
core/extrema_trainer.py
View File

@@ -423,13 +423,13 @@ class ExtremaTrainer:
self.context_data[symbol].last_update = datetime.now()
results[symbol] = True
logger.info(f"✅ Loaded {len(context_data)} 1m candles for {symbol} context")
logger.info(f" Loaded {len(context_data)} 1m candles for {symbol} context")
else:
results[symbol] = False
logger.warning(f"❌ No 1m context data available for {symbol}")
logger.warning(f" No 1m context data available for {symbol}")
except Exception as e:
logger.error(f"❌ Error loading context data for {symbol}: {e}")
logger.error(f" Error loading context data for {symbol}: {e}")
results[symbol] = False
except Exception as e:

12
core/overnight_training_coordinator.py
View File

@@ -110,11 +110,11 @@ class OvernightTrainingCoordinator:
logger.info("🌙 OVERNIGHT TRAINING SESSION STARTED")
logger.info("=" * 60)
logger.info("Features enabled:")
logger.info("✅ CNN training on signal changes")
logger.info("✅ COB RL training on market microstructure")
logger.info("✅ Trade execution and recording")
logger.info("✅ Performance tracking and statistics")
logger.info("✅ Model checkpointing")
logger.info(" CNN training on signal changes")
logger.info(" COB RL training on market microstructure")
logger.info(" Trade execution and recording")
logger.info(" Performance tracking and statistics")
logger.info(" Model checkpointing")
logger.info("=" * 60)
def stop_overnight_training(self):
@@ -253,7 +253,7 @@ class OvernightTrainingCoordinator:
status = "EXECUTED" if signal_record.executed else "SIGNAL_ONLY"
logger.info(f"[{status}] {symbol} {decision.action} "
f"(conf: {decision.confidence:.3f}, "
f"training: {'✅' if signal_record.training_triggered else '❌'}, "
f"training: {'' if signal_record.training_triggered else ''}, "
f"pnl: {signal_record.trade_pnl:.2f if signal_record.trade_pnl else 'N/A'})")
except Exception as e:

14
core/realtime_prediction_loop.py
View File

@@ -49,7 +49,7 @@ class RealtimePredictionLoop:
async def start(self):
"""Start the continuous prediction loop"""
self.running = True
logger.info("🔄 Starting Real-Time Prediction Loop")
logger.info(" Starting Real-Time Prediction Loop")
# Start prediction tasks for each symbol
symbols = self.orchestrator.config.get('symbols', ['ETH/USDT', 'BTC/USDT'])
@@ -67,7 +67,7 @@ class RealtimePredictionLoop:
async def _prediction_loop_for_symbol(self, symbol: str):
"""Run prediction loop for a specific symbol"""
logger.info(f"🔄 Prediction loop started for {symbol}")
logger.info(f" Prediction loop started for {symbol}")
while self.running:
try:
@@ -194,10 +194,10 @@ class RealtimePredictionLoop:
if cnn_input and cnn_input.data_quality_score > 0.5:
cnn_data = self.unified_data_interface.get_model_specific_input(cnn_input, 'cnn')
if cnn_data is not None:
# ✅ THIS IS WHERE model.predict() GETS CALLED WITH CORRECT DATA!
# THIS IS WHERE model.predict() GETS CALLED WITH CORRECT DATA!
cnn_prediction = self.orchestrator.cnn_model.predict(cnn_data)
predictions['cnn'] = cnn_prediction
logger.info(f"✅ CNN prediction for {symbol}: {cnn_prediction} (quality: {cnn_input.data_quality_score:.2f})")
logger.info(f" CNN prediction for {symbol}: {cnn_prediction} (quality: {cnn_input.data_quality_score:.2f})")
except Exception as e:
logger.error(f"CNN prediction error for {symbol}: {e}")
@@ -215,7 +215,7 @@ class RealtimePredictionLoop:
'action': action,
'action_name': ['SELL', 'HOLD', 'BUY'][action]
}
logger.info(f"✅ DQN prediction for {symbol}: {predictions['dqn']['action_name']} (quality: {dqn_input.data_quality_score:.2f})")
logger.info(f" DQN prediction for {symbol}: {predictions['dqn']['action_name']} (quality: {dqn_input.data_quality_score:.2f})")
except Exception as e:
logger.error(f"DQN prediction error for {symbol}: {e}")
@@ -229,7 +229,7 @@ class RealtimePredictionLoop:
if cob_data is not None and hasattr(self.orchestrator.cob_rl_model, 'predict'):
cob_prediction = self.orchestrator.cob_rl_model.predict(cob_data)
predictions['cob_rl'] = cob_prediction
logger.info(f"✅ COB RL prediction for {symbol}: {cob_prediction} (quality: {cob_input.data_quality_score:.2f})")
logger.info(f" COB RL prediction for {symbol}: {cob_prediction} (quality: {cob_input.data_quality_score:.2f})")
except Exception as e:
logger.error(f"COB RL prediction error for {symbol}: {e}")
@@ -243,7 +243,7 @@ class RealtimePredictionLoop:
if transformer_data is not None and hasattr(self.orchestrator.transformer_model, 'predict'):
transformer_prediction = self.orchestrator.transformer_model.predict(transformer_data)
predictions['transformer'] = transformer_prediction
logger.info(f"✅ Transformer prediction for {symbol}: {transformer_prediction} (quality: {transformer_input.data_quality_score:.2f})")
logger.info(f" Transformer prediction for {symbol}: {transformer_prediction} (quality: {transformer_input.data_quality_score:.2f})")
except Exception as e:
logger.error(f"Transformer prediction error for {symbol}: {e}")

2
core/trading_executor.py
View File

@@ -811,7 +811,7 @@ class TradingExecutor:
self.min_profitability_multiplier,
self.profitability_reward_multiplier - self.profitability_adjustment_step
)
logger.info(f"⚠️ SUCCESS RATE LOW ({success_rate:.1%}) - Decreased profitability multiplier: {old_multiplier:.1f} → {self.profitability_reward_multiplier:.1f}")
logger.info(f" SUCCESS RATE LOW ({success_rate:.1%}) - Decreased profitability multiplier: {old_multiplier:.1f} → {self.profitability_reward_multiplier:.1f}")
else:
logger.debug(f"Success rate {success_rate:.1%} in acceptable range - keeping multiplier at {self.profitability_reward_multiplier:.1f}")

4
core/unified_data_provider_extension.py
View File

@@ -96,7 +96,7 @@ class UnifiedDataProviderExtension:
logger.info("✓ Ingestion pipeline started")
self._initialized = True
logger.info("✅ Unified storage system initialized successfully")
logger.info(" Unified storage system initialized successfully")
return True
@@ -119,7 +119,7 @@ class UnifiedDataProviderExtension:
logger.info("✓ Database connection closed")
self._initialized = False
logger.info("✅ Unified storage system shutdown complete")
logger.info(" Unified storage system shutdown complete")
except Exception as e:
logger.error(f"Error shutting down unified storage: {e}")

4
core/unified_queryable_storage.py
View File

@@ -60,7 +60,7 @@ class UnifiedQueryableStorage:
self.backend = get_timescale_storage(timescale_connection_string)
if self.backend:
self.backend_type = "timescale"
logger.info("✅ Using TimescaleDB for queryable storage")
logger.info(" Using TimescaleDB for queryable storage")
except Exception as e:
logger.warning(f"TimescaleDB not available: {e}")
@@ -69,7 +69,7 @@ class UnifiedQueryableStorage:
try:
self.backend = SQLiteQueryableStorage(sqlite_path)
self.backend_type = "sqlite"
logger.info("✅ Using SQLite for queryable storage (TimescaleDB fallback)")
logger.info(" Using SQLite for queryable storage (TimescaleDB fallback)")
except Exception as e:
logger.error(f"Failed to initialize SQLite storage: {e}")
raise Exception("No queryable storage backend available")

34
docs/CURRENT_RL_INPUT_ANALYSIS.md
View File

@@ -26,14 +26,14 @@ def _market_state_to_rl_state(self, market_state: MarketState) -> np.ndarray:
**Total Current Input: ~100 basic features (CRITICALLY INSUFFICIENT)**
### What's Missing from Current Implementation:
- ❌ **300s of raw tick data** (0 features vs required 3000+ features)
- ❌ **Multi-timeframe OHLCV data** (4 basic prices vs required 9600+ features)
- ❌ **BTC reference data** (0 features vs required 2400+ features)
- ❌ **CNN hidden layer features** (0 features vs required 512 features)
- ❌ **CNN predictions** (0 features vs required 16 features)
- ❌ **Pivot point data** (0 features vs required 250+ features)
- ❌ **Momentum detection from ticks** (completely missing)
- ❌ **Market regime analysis** (basic vs sophisticated analysis)
- **300s of raw tick data** (0 features vs required 3000+ features)
- **Multi-timeframe OHLCV data** (4 basic prices vs required 9600+ features)
- **BTC reference data** (0 features vs required 2400+ features)
- **CNN hidden layer features** (0 features vs required 512 features)
- **CNN predictions** (0 features vs required 16 features)
- **Pivot point data** (0 features vs required 250+ features)
- **Momentum detection from ticks** (completely missing)
- **Market regime analysis** (basic vs sophisticated analysis)
## What Dashboard Currently Shows
@@ -52,37 +52,37 @@ This shows the data is being **collected** but **NOT being fed to the RL model**
### ETH Data Requirements:
1. **300s max of raw ticks data** → ~3000 features
- Important for detecting single big moves and momentum
- Currently: 0 features ❌
- Currently: 0 features
2. **300s of 1s OHLCV data (5 min)** → 2400 features
- 300 bars × 8 features (OHLC + volume + indicators)
- Currently: 0 features ❌
- Currently: 0 features
3. **300 OHLCV + indicators bars for each timeframe** → 7200 features
- 1m: 300 bars × 8 features = 2400
- 1h: 300 bars × 8 features = 2400
- 1d: 300 bars × 8 features = 2400
- Currently: ~4 basic price features ❌
- Currently: ~4 basic price features
### BTC Reference Data:
4. **BTC data for all timeframes** → 2400 features
- Same structure as ETH for correlation analysis
- Currently: 0 features ❌
- Currently: 0 features
### CNN Integration:
5. **CNN hidden layer features** → 512 features
- Last hidden layers where patterns are learned
- Currently: 0 features ❌
- Currently: 0 features
6. **CNN predictions for each timeframe** → 16 features
- 1s, 1m, 1h, 1d predictions (4 timeframes × 4 outputs)
- Currently: 0 features ❌
- Currently: 0 features
### Pivot Points:
7. **Williams Market Structure pivot points** → 250+ features
- 5-level recursive pivot point calculation
- Standard pivot points for all timeframes
- Currently: 0 features ❌
- Currently: 0 features
## Total Required vs Current
@@ -113,12 +113,12 @@ This explains why RL performance may be poor - the model is essentially "blind"
## Solution Implementation Status
✅ **Already Created**:
**Already Created**:
- `training/enhanced_rl_state_builder.py` - Implements comprehensive state building
- `training/williams_market_structure.py` - Williams pivot point system
- `docs/RL_TRAINING_AUDIT_AND_IMPROVEMENTS.md` - Complete improvement plan
⚠️ **Next Steps**:
**Next Steps**:
1. Integrate the enhanced state builder into the current RL training pipeline
2. Update MarketState class to include all required data
3. Connect tick cache and OHLCV data to state builder

38
docs/ENHANCED_RL_REAL_DATA_INTEGRATION.md
View File

@@ -34,7 +34,7 @@ comprehensive_state = self.state_builder.build_rl_state(
## Real Data Sources Integration
### 1. Tick Data (300s Window) ✅
### 1. Tick Data (300s Window)
**Source**: Your dashboard's "Tick Cache: 129 ticks"
```python
def _get_recent_tick_data_for_rl(self, symbol: str, seconds: int = 300):
@@ -43,7 +43,7 @@ def _get_recent_tick_data_for_rl(self, symbol: str, seconds: int = 300):
# Converts to RL format with momentum detection
```
### 2. Multi-timeframe OHLCV ✅
### 2. Multi-timeframe OHLCV
**Source**: Your dashboard's "1s Bars: 128 bars" + historical data
```python
def _get_multiframe_ohlcv_for_rl(self, symbol: str):
@@ -51,21 +51,21 @@ def _get_multiframe_ohlcv_for_rl(self, symbol: str):
# Gets real OHLCV data with technical indicators (RSI, MACD, BB, etc.)
```
### 3. BTC Reference Data ✅
### 3. BTC Reference Data
**Source**: Same data provider, BTC/USDT symbol
```python
btc_reference_data = self._get_multiframe_ohlcv_for_rl('BTC/USDT')
# Provides correlation analysis for ETH decisions
```
### 4. Williams Market Structure ✅
### 4. Williams Market Structure
**Source**: Calculated from real 1m OHLCV data
```python
pivot_data = self.williams_structure.calculate_recursive_pivot_points(ohlc_array)
# Implements your specified 5-level recursive pivot system
```
### 5. CNN Integration Framework ✅
### 5. CNN Integration Framework
**Ready for**: CNN hidden features and predictions
```python
def _get_cnn_features_for_rl(self, symbol: str):
@@ -75,21 +75,21 @@ def _get_cnn_features_for_rl(self, symbol: str):
## Files Modified/Created
### 1. Enhanced RL Trainer (`training/enhanced_rl_trainer.py`) ✅
### 1. Enhanced RL Trainer (`training/enhanced_rl_trainer.py`)
- **Replaced** mock `_market_state_to_rl_state()` with comprehensive state building
- **Integrated** with EnhancedRLStateBuilder (~13,400 features)
- **Connected** to real data sources (ticks, OHLCV, BTC reference)
- **Added** Williams pivot point calculation
- **Enhanced** agent initialization with larger state space (1024 hidden units)
### 2. Enhanced Orchestrator (`core/enhanced_orchestrator.py`) ✅
### 2. Enhanced Orchestrator (`core/enhanced_orchestrator.py`)
- **Expanded** MarketState class with comprehensive data fields
- **Added** real tick data extraction methods
- **Implemented** multi-timeframe OHLCV processing with technical indicators
- **Integrated** market microstructure analysis
- **Added** CNN feature extraction framework
### 3. Comprehensive Launcher (`run_enhanced_rl_training.py`) ✅
### 3. Comprehensive Launcher (`run_enhanced_rl_training.py`)
- **Created** complete training system launcher
- **Implements** real-time data collection and verification
- **Provides** comprehensive training loop with real market states
@@ -122,7 +122,7 @@ Stream: LIVE + Technical Indic. + CNN features + Pivots
## New Capabilities Unlocked
### 1. Momentum Detection 🚀
### 1. Momentum Detection
- **Real tick-level analysis** for detecting single big moves
- **Volume-weighted price momentum** from 300s of tick data
- **Market microstructure patterns** (order flow, tick frequency)
@@ -188,16 +188,16 @@ The system includes comprehensive data quality monitoring:
## Integration Status
✅ **COMPLETE**: Real tick data integration (300s window)
✅ **COMPLETE**: Multi-timeframe OHLCV processing
✅ **COMPLETE**: BTC reference data integration
✅ **COMPLETE**: Williams Market Structure implementation
✅ **COMPLETE**: Technical indicators (RSI, MACD, BB, ATR)
✅ **COMPLETE**: Market microstructure analysis
✅ **COMPLETE**: Comprehensive state building (~13,400 features)
✅ **COMPLETE**: Real-time training loop
✅ **COMPLETE**: Data quality monitoring
⚠️ **FRAMEWORK READY**: CNN hidden feature extraction (when CNN models available)
**COMPLETE**: Real tick data integration (300s window)
**COMPLETE**: Multi-timeframe OHLCV processing
**COMPLETE**: BTC reference data integration
**COMPLETE**: Williams Market Structure implementation
**COMPLETE**: Technical indicators (RSI, MACD, BB, ATR)
**COMPLETE**: Market microstructure analysis
**COMPLETE**: Comprehensive state building (~13,400 features)
**COMPLETE**: Real-time training loop
**COMPLETE**: Data quality monitoring
**FRAMEWORK READY**: CNN hidden feature extraction (when CNN models available)
## Performance Impact Expected

66
docs/UNIFIED_STORAGE_COMPLETE.md
View File

@@ -4,9 +4,9 @@
The unified data storage system has been successfully implemented and integrated into the existing DataProvider.
## ✅ Completed Tasks (8 out of 10)
## Completed Tasks (8 out of 10)
### Task 1: TimescaleDB Schema and Infrastructure ✅
### Task 1: TimescaleDB Schema and Infrastructure
**Files:**
- `core/unified_storage_schema.py` - Schema manager with migrations
- `scripts/setup_unified_storage.py` - Automated setup script
@@ -19,7 +19,7 @@ The unified data storage system has been successfully implemented and integrated
- Compression policies (>80% compression)
- Retention policies (30 days to 2 years)
### Task 2: Data Models and Validation ✅
### Task 2: Data Models and Validation
**Files:**
- `core/unified_data_models.py` - Data structures
- `core/unified_data_validator.py` - Validation logic
@@ -30,7 +30,7 @@ The unified data storage system has been successfully implemented and integrated
- `OHLCVCandle`, `TradeEvent` - Individual data types
- Comprehensive validation and sanitization
### Task 3: Cache Layer ✅
### Task 3: Cache Layer
**Files:**
- `core/unified_cache_manager.py` - In-memory caching
@@ -41,7 +41,7 @@ The unified data storage system has been successfully implemented and integrated
- Automatic eviction
- Statistics tracking
### Task 4: Database Connection and Query Layer ✅
### Task 4: Database Connection and Query Layer
**Files:**
- `core/unified_database_manager.py` - Connection pool and queries
@@ -52,7 +52,7 @@ The unified data storage system has been successfully implemented and integrated
- <100ms query latency
- Multi-timeframe support
### Task 5: Data Ingestion Pipeline ✅
### Task 5: Data Ingestion Pipeline
**Files:**
- `core/unified_ingestion_pipeline.py` - Real-time ingestion
@@ -63,7 +63,7 @@ The unified data storage system has been successfully implemented and integrated
- >1000 ops/sec throughput
- Error handling and retry logic
### Task 6: Unified Data Provider API ✅
### Task 6: Unified Data Provider API
**Files:**
- `core/unified_data_provider_extension.py` - Main API
@@ -74,10 +74,10 @@ The unified data storage system has been successfully implemented and integrated
- Order book data access
- Statistics tracking
### Task 7: Data Migration System ✅
### Task 7: Data Migration System
**Status:** Skipped (decided to drop existing Parquet data)
### Task 8: Integration with Existing DataProvider ✅
### Task 8: Integration with Existing DataProvider
**Files:**
- `core/data_provider.py` - Updated with unified storage methods
- `docs/UNIFIED_STORAGE_INTEGRATION.md` - Integration guide
@@ -115,27 +115,27 @@ The unified data storage system has been successfully implemented and integrated
└──────────────┘ └──────────────┘
```
## 🚀 Key Features
## Key Features
### Performance
- ✅ Cache reads: <10ms
- ✅ Database queries: <100ms
- ✅ Ingestion: >1000 ops/sec
- ✅ Compression: >80%
- Cache reads: <10ms
- Database queries: <100ms
- Ingestion: >1000 ops/sec
- Compression: >80%
### Reliability
- ✅ Data validation
- ✅ Error handling
- ✅ Health monitoring
- ✅ Statistics tracking
- ✅ Automatic reconnection
- Data validation
- Error handling
- Health monitoring
- Statistics tracking
- Automatic reconnection
### Usability
- ✅ Single endpoint for all data
- ✅ Automatic routing (cache vs database)
- ✅ Type-safe interfaces
- ✅ Backward compatible
- ✅ Easy to integrate
- Single endpoint for all data
- Automatic routing (cache vs database)
- Type-safe interfaces
- Backward compatible
- Easy to integrate
## 📝 Quick Start
@@ -302,9 +302,9 @@ print(f"Ingestion rate: {stats['ingestion']['total_ingested']}")
### Check Health
```python
if data_provider.is_unified_storage_enabled():
print("✅ Unified storage is running")
print(" Unified storage is running")
else:
print("❌ Unified storage is not enabled")
print(" Unified storage is not enabled")
```
## 🚧 Remaining Tasks (Optional)
@@ -323,12 +323,12 @@ else:
## 🎉 Success Metrics
✅ **Completed**: 8 out of 10 major tasks (80%)
✅ **Core Functionality**: 100% complete
✅ **Integration**: Seamless with existing code
✅ **Performance**: Meets all targets
✅ **Documentation**: Comprehensive guides
✅ **Examples**: Working code samples
**Completed**: 8 out of 10 major tasks (80%)
**Core Functionality**: 100% complete
**Integration**: Seamless with existing code
**Performance**: Meets all targets
**Documentation**: Comprehensive guides
**Examples**: Working code samples
## 🙏 Next Steps
@@ -349,7 +349,7 @@ For issues or questions:
---
**Status**: ✅ Production Ready
**Status**: Production Ready
**Version**: 1.0.0
**Last Updated**: 2024
**Completion**: 80% (8/10 tasks)

14
docs/UNIFIED_STORAGE_INTEGRATION.md
View File

@@ -6,11 +6,11 @@ The unified storage system has been integrated into the existing `DataProvider`
## Key Features
✅ **Single Endpoint**: One method for all data access
✅ **Automatic Routing**: Cache for real-time, database for historical
✅ **Backward Compatible**: All existing methods still work
✅ **Opt-In**: Only enabled when explicitly initialized
✅ **Fast**: <10ms cache reads, <100ms database queries
**Single Endpoint**: One method for all data access
**Automatic Routing**: Cache for real-time, database for historical
**Backward Compatible**: All existing methods still work
**Opt-In**: Only enabled when explicitly initialized
**Fast**: <10ms cache reads, <100ms database queries
## Quick Start
@@ -27,9 +27,9 @@ data_provider = DataProvider()
async def setup():
success = await data_provider.enable_unified_storage()
if success:
print("✅ Unified storage enabled!")
print(" Unified storage enabled!")
else:
print("❌ Failed to enable unified storage")
print(" Failed to enable unified storage")
asyncio.run(setup())
```

12
docs/fifo_queue_system.md
View File

@@ -250,12 +250,12 @@ python test_fifo_queues.py
```
**Test Coverage**:
- ✅ FIFO queue operations (add, retrieve, status)
- ✅ Data queue filling with multiple timeframes
- ✅ BaseDataInput building from queues
- ✅ Consistent feature vector size (always 7850)
- ✅ Thread safety under concurrent access
- ✅ Minimum data requirement validation
- FIFO queue operations (add, retrieve, status)
- Data queue filling with multiple timeframes
- BaseDataInput building from queues
- Consistent feature vector size (always 7850)
- Thread safety under concurrent access
- Minimum data requirement validation
## Monitoring

4
examples/report_crawler_example.py
View File

@@ -73,9 +73,9 @@ def generate_reports_example():
print(f"\nBatch Results:")
for symbol, data in results.items():
if data:
print(f"✅ {symbol}: ${data.current_price:,.2f} (RSI: {data.current_rsi_7:.1f})")
print(f" {symbol}: ${data.current_price:,.2f} (RSI: {data.current_rsi_7:.1f})")
else:
print(f"❌ {symbol}: Failed to get data")
print(f" {symbol}: Failed to get data")
return True

8
examples/unified_storage_example.py
View File

@@ -30,10 +30,10 @@ async def example_realtime_data():
success = await data_provider.enable_unified_storage()
if not success:
print("❌ Failed to enable unified storage")
print(" Failed to enable unified storage")
return
print("✅ Unified storage enabled")
print(" Unified storage enabled")
# Get latest real-time data
print("\n2. Getting latest real-time data...")
@@ -261,11 +261,11 @@ async def main():
await example_statistics()
print("\n" + "="*60)
print("✅ All examples completed successfully!")
print(" All examples completed successfully!")
print("="*60 + "\n")
except Exception as e:
print(f"\n❌ Error running examples: {e}")
print(f"\n Error running examples: {e}")
import traceback
traceback.print_exc()

6
readme.md
View File

@@ -92,12 +92,12 @@ gogo2/
## Data Sources
### ✅ Approved Sources
### Approved Sources
- Binance API (real-time and historical)
- Cached real market data
- TimescaleDB with real data
### ❌ Prohibited Sources
### Prohibited Sources
- Synthetic data generation
- Random data simulation
- Mock market conditions
@@ -228,4 +228,4 @@ This project is for educational and research purposes. Use real market data resp
---
**⚠️ REMEMBER: This system's integrity depends on using only real market data. No exceptions.**
** REMEMBER: This system's integrity depends on using only real market data. No exceptions.**

16
test_amd_gpu.sh
View File

@@ -7,32 +7,32 @@ echo ""
# Check if AMD GPU devices are available
echo "Checking AMD GPU devices..."
if [[ -e /dev/kfd ]]; then
echo "✅ /dev/kfd (AMD GPU compute) is available"
echo " /dev/kfd (AMD GPU compute) is available"
else
echo "❌ /dev/kfd not found - AMD GPU compute not available"
echo " /dev/kfd not found - AMD GPU compute not available"
fi
if [[ -e /dev/dri/renderD128 ]] || [[ -e /dev/dri/card0 ]]; then
echo "✅ /dev/dri (AMD GPU graphics) is available"
echo " /dev/dri (AMD GPU graphics) is available"
else
echo "❌ /dev/dri not found - AMD GPU graphics not available"
echo " /dev/dri not found - AMD GPU graphics not available"
fi
echo ""
echo "Checking user groups..."
if groups | grep -q video; then
echo "✅ User is in 'video' group for GPU access"
echo " User is in 'video' group for GPU access"
else
echo "⚠️ User is not in 'video' group - may need: sudo usermod -aG video $USER"
echo " User is not in 'video' group - may need: sudo usermod -aG video $USER"
fi
echo ""
echo "Testing Docker with AMD GPU..."
# Test if docker can access AMD GPU devices
if docker run --rm --device /dev/kfd:/dev/kfd --device /dev/dri:/dev/dri alpine ls /dev/kfd /dev/dri 2>/dev/null | grep -q kfd; then
echo "✅ Docker can access AMD GPU devices"
echo " Docker can access AMD GPU devices"
else
echo "❌ Docker cannot access AMD GPU devices"
echo " Docker cannot access AMD GPU devices"
echo " Try: sudo chmod 666 /dev/kfd /dev/dri/*"
fi

12
test_direction_filter.py
View File

@@ -43,9 +43,9 @@ def test_filter_by_time_range():
print(f"Last: {result.index[-1]} (should be minute 49)")
if result.index[-1] < end_time:
print("✅ PASS: Last candle is before end_time")
print(" PASS: Last candle is before end_time")
else:
print("❌ FAIL: Last candle is NOT before end_time")
print(" FAIL: Last candle is NOT before end_time")
# Test 2: Direction 'after'
print("\n" + "-" * 80)
@@ -62,9 +62,9 @@ def test_filter_by_time_range():
print(f"Last: {result.index[-1]} (should be minute 60)")
if result.index[0] > start_time:
print("✅ PASS: First candle is after start_time")
print(" PASS: First candle is after start_time")
else:
print("❌ FAIL: First candle is NOT after start_time")
print(" FAIL: First candle is NOT after start_time")
# Test 3: Direction 'latest' (default)
print("\n" + "-" * 80)
@@ -77,9 +77,9 @@ def test_filter_by_time_range():
print(f"Last: {result.index[-1]} (should be minute 99)")
if result.index[-1] == df.index[-1]:
print("✅ PASS: Got most recent candles")
print(" PASS: Got most recent candles")
else:
print("❌ FAIL: Did NOT get most recent candles")
print(" FAIL: Did NOT get most recent candles")
print("\n" + "=" * 80)
print("All Tests Complete")

42
test_duckdb_storage.py
View File

@@ -33,11 +33,11 @@ try:
from core.duckdb_storage import DuckDBStorage
storage = DuckDBStorage()
print(f"✅ DuckDB initialized: {storage.db_path}")
print(f" DuckDB initialized: {storage.db_path}")
print(f" Parquet directory: {storage.parquet_dir}")
except Exception as e:
print(f"❌ FAIL: {e}")
print(f" FAIL: {e}")
import traceback
traceback.print_exc()
sys.exit(1)
@@ -54,13 +54,13 @@ try:
init_time = time.time() - start_time
if data_provider.duckdb_storage:
print(f"✅ DataProvider has DuckDB storage")
print(f" DataProvider has DuckDB storage")
print(f" Initialization time: {init_time:.2f}s")
else:
print(f"⚠️ DataProvider missing DuckDB storage")
print(f" DataProvider missing DuckDB storage")
except Exception as e:
print(f"❌ FAIL: {e}")
print(f" FAIL: {e}")
import traceback
traceback.print_exc()
@@ -83,13 +83,13 @@ try:
fetch_time = time.time() - start_time
if df is not None and not df.empty:
print(f"✅ Fetched {len(df)} candles in {fetch_time:.2f}s")
print(f" Fetched {len(df)} candles in {fetch_time:.2f}s")
print(f" Data automatically stored in DuckDB")
else:
print(f"⚠️ No data fetched")
print(f" No data fetched")
except Exception as e:
print(f"❌ FAIL: {e}")
print(f" FAIL: {e}")
import traceback
traceback.print_exc()
@@ -110,13 +110,13 @@ try:
query_time = time.time() - start_time
if df is not None and not df.empty:
print(f"✅ Retrieved {len(df)} candles in {query_time:.3f}s")
print(f" Retrieved {len(df)} candles in {query_time:.3f}s")
print(f" Query speed: {query_time*1000:.1f}ms")
else:
print(f"⚠️ No data in DuckDB yet")
print(f" No data in DuckDB yet")
except Exception as e:
print(f"❌ FAIL: {e}")
print(f" FAIL: {e}")
import traceback
traceback.print_exc()
@@ -133,15 +133,15 @@ try:
""")
if not result.empty:
print(f"✅ SQL query successful")
print(f" SQL query successful")
print("\nCache metadata:")
for _, row in result.iterrows():
print(f" {row['symbol']} {row['timeframe']}: {row['candle_count']:,} candles")
else:
print(f"⚠️ No metadata yet")
print(f" No metadata yet")
except Exception as e:
print(f"❌ FAIL: {e}")
print(f" FAIL: {e}")
import traceback
traceback.print_exc()
@@ -161,10 +161,10 @@ try:
for stat in ohlcv_stats:
print(f" {stat['symbol']} {stat['timeframe']}: {stat['candle_count']:,} candles")
print(f"✅ Statistics retrieved successfully")
print(f" Statistics retrieved successfully")
except Exception as e:
print(f"❌ FAIL: {e}")
print(f" FAIL: {e}")
import traceback
traceback.print_exc()
@@ -178,15 +178,15 @@ try:
ann_manager = AnnotationManager()
if ann_manager.duckdb_storage:
print(f"✅ Annotation manager has DuckDB storage")
print(f" Annotation manager has DuckDB storage")
else:
print(f"⚠️ Annotation manager missing DuckDB storage")
print(f" Annotation manager missing DuckDB storage")
annotations = ann_manager.get_annotations()
print(f" Existing annotations: {len(annotations)}")
except Exception as e:
print(f"❌ FAIL: {e}")
print(f" FAIL: {e}")
import traceback
traceback.print_exc()
@@ -195,7 +195,7 @@ print("\n" + "=" * 80)
print("TEST SUMMARY")
print("=" * 80)
print("\n✅ DuckDB Integration:")
print("\n DuckDB Integration:")
print(" - Storage initialized: WORKING")
print(" - DataProvider integration: WORKING")
print(" - Data storage: WORKING")
@@ -224,5 +224,5 @@ print(" 3. Query data with SQL for analysis")
print(" 4. Enjoy unified storage!")
print("\n" + "=" * 80)
print("✅ ALL TESTS COMPLETED")
print(" ALL TESTS COMPLETED")
print("=" * 80)

24
test_infinite_scroll_backend.py
View File

@@ -39,10 +39,10 @@ def test_backend_data_loading():
)
if initial_df is None or initial_df.empty:
print("❌ FAILED: No initial data loaded")
print(" FAILED: No initial data loaded")
return
print(f"✅ Loaded {len(initial_df)} initial candles")
print(f" Loaded {len(initial_df)} initial candles")
print(f" First timestamp: {initial_df.index[0]}")
print(f" Last timestamp: {initial_df.index[-1]}")
@@ -59,20 +59,20 @@ def test_backend_data_loading():
)
if older_df is None or older_df.empty:
print("❌ FAILED: No older data loaded")
print(" FAILED: No older data loaded")
print(" This might mean:")
print(" - No data exists before this timestamp in DuckDB")
print(" - The query is not working correctly")
else:
print(f"✅ Loaded {len(older_df)} older candles")
print(f" Loaded {len(older_df)} older candles")
print(f" First timestamp: {older_df.index[0]}")
print(f" Last timestamp: {older_df.index[-1]}")
# Check if older data is actually older
if older_df.index[-1] < first_timestamp:
print(f"✅ Data is correctly older (last older candle: {older_df.index[-1]} < first initial: {first_timestamp})")
print(f" Data is correctly older (last older candle: {older_df.index[-1]} < first initial: {first_timestamp})")
else:
print(f"❌ WARNING: Data is NOT older! Last older: {older_df.index[-1]} >= first initial: {first_timestamp}")
print(f" WARNING: Data is NOT older! Last older: {older_df.index[-1]} >= first initial: {first_timestamp}")
# Test 3: Load newer data (after last timestamp)
print(f"\n5. Loading newer data AFTER {initial_df.index[-1]}...")
@@ -87,17 +87,17 @@ def test_backend_data_loading():
)
if newer_df is None or newer_df.empty:
print("❌ No newer data loaded (this is expected if we're at the latest data)")
print(" No newer data loaded (this is expected if we're at the latest data)")
else:
print(f"✅ Loaded {len(newer_df)} newer candles")
print(f" Loaded {len(newer_df)} newer candles")
print(f" First timestamp: {newer_df.index[0]}")
print(f" Last timestamp: {newer_df.index[-1]}")
# Check if newer data is actually newer
if newer_df.index[0] > last_timestamp:
print(f"✅ Data is correctly newer (first newer candle: {newer_df.index[0]} > last initial: {last_timestamp})")
print(f" Data is correctly newer (first newer candle: {newer_df.index[0]} > last initial: {last_timestamp})")
else:
print(f"❌ WARNING: Data is NOT newer! First newer: {newer_df.index[0]} <= last initial: {last_timestamp}")
print(f" WARNING: Data is NOT newer! First newer: {newer_df.index[0]} <= last initial: {last_timestamp}")
# Test 4: Check DuckDB directly
print(f"\n6. Checking DuckDB storage directly...")
@@ -110,7 +110,7 @@ def test_backend_data_loading():
print(f" Total candles in DuckDB: {total_count}")
if total_count == 0:
print(" ❌ No data in DuckDB! Need to fetch from API first.")
print(" No data in DuckDB! Need to fetch from API first.")
else:
# Get time range
query = """
@@ -129,7 +129,7 @@ def test_backend_data_loading():
print(f" Time range: {min_ts} to {max_ts}")
print(f" Duration: {max_ts - min_ts}")
else:
print(" ❌ DuckDB storage not available")
print(" DuckDB storage not available")
print("\n" + "=" * 80)
print("Test Complete")

10
test_pivot_levels.py
View File

@@ -37,10 +37,10 @@ def test_pivot_levels():
pivot_levels = data_provider.get_williams_pivot_levels(symbol, base_timeframe='1m', limit=5000)
if not pivot_levels:
logger.error(f"❌ NO PIVOT LEVELS for {symbol}")
logger.error(f" NO PIVOT LEVELS for {symbol}")
return False
logger.info(f"✅ Found {len(pivot_levels)} pivot levels")
logger.info(f" Found {len(pivot_levels)} pivot levels")
for level_num in sorted(pivot_levels.keys()):
trend_level = pivot_levels[level_num]
@@ -61,10 +61,10 @@ def test_pivot_levels():
actual_levels = set(pivot_levels.keys())
if expected_levels.issubset(actual_levels):
logger.info("✅ ALL 5 PIVOT LEVELS PRESENT!")
logger.info(" ALL 5 PIVOT LEVELS PRESENT!")
else:
missing = expected_levels - actual_levels
logger.warning(f"❌ MISSING LEVELS: {missing}")
logger.warning(f" MISSING LEVELS: {missing}")
return True
@@ -79,6 +79,6 @@ if __name__ == "__main__":
if success:
print("\n🎉 Pivot levels test completed!")
else:
print("\n❌ Pivot levels test failed!")
print("\n Pivot levels test failed!")
sys.exit(1)

30
test_training.py
View File

@@ -24,13 +24,13 @@ def test_training():
# Step 1: Initialize components
print("\n1. Initializing components...")
data_provider = DataProvider()
print(" ✅ DataProvider initialized")
print(" DataProvider initialized")
orchestrator = TradingOrchestrator(
data_provider=data_provider,
enhanced_rl_training=True
)
print(" ✅ Orchestrator initialized")
print(" Orchestrator initialized")
# Step 2: Initialize ML models
print("\n2. Initializing ML models...")
@@ -40,25 +40,25 @@ def test_training():
available_models = []
if hasattr(orchestrator, 'rl_agent') and orchestrator.rl_agent:
available_models.append('DQN')
print(" ✅ DQN model available")
print(" DQN model available")
if hasattr(orchestrator, 'cnn_model') and orchestrator.cnn_model:
available_models.append('CNN')
print(" ✅ CNN model available")
print(" CNN model available")
if hasattr(orchestrator, 'primary_transformer') and orchestrator.primary_transformer:
available_models.append('Transformer')
print(" ✅ Transformer model available")
print(" Transformer model available")
# Check if trainer is available
if hasattr(orchestrator, 'primary_transformer_trainer') and orchestrator.primary_transformer_trainer:
trainer = orchestrator.primary_transformer_trainer
print(f" ✅ Transformer trainer available: {type(trainer).__name__}")
print(f" Transformer trainer available: {type(trainer).__name__}")
# List available methods
methods = [m for m in dir(trainer) if not m.startswith('_') and callable(getattr(trainer, m))]
print(f" 📋 Trainer methods: {', '.join(methods[:10])}...")
if not available_models:
print(" ❌ No models available!")
print(" No models available!")
return
print(f"\n Available models: {', '.join(available_models)}")
@@ -66,16 +66,16 @@ def test_training():
# Step 3: Initialize training adapter
print("\n3. Initializing training adapter...")
training_adapter = RealTrainingAdapter(orchestrator, data_provider)
print(" ✅ Training adapter initialized")
print(" Training adapter initialized")
# Step 4: Load test cases
print("\n4. Loading test cases...")
annotation_manager = AnnotationManager()
test_cases = annotation_manager.get_all_test_cases()
print(f" ✅ Loaded {len(test_cases)} test cases")
print(f" Loaded {len(test_cases)} test cases")
if len(test_cases) == 0:
print(" ⚠️ No test cases available - create some annotations first!")
print(" No test cases available - create some annotations first!")
return
# Step 5: Start training
@@ -87,7 +87,7 @@ def test_training():
model_name='Transformer',
test_cases=test_cases
)
print(f" ✅ Training started: {training_id}")
print(f" Training started: {training_id}")
# Step 6: Monitor training progress
print("\n6. Monitoring training progress...")
@@ -98,25 +98,25 @@ def test_training():
progress = training_adapter.get_training_progress(training_id)
if progress['status'] == 'completed':
print(f"\n ✅ Training completed!")
print(f"\n Training completed!")
print(f" Final loss: {progress['final_loss']:.6f}")
print(f" Accuracy: {progress['accuracy']:.2%}")
print(f" Duration: {progress['duration_seconds']:.2f}s")
break
elif progress['status'] == 'failed':
print(f"\n ❌ Training failed!")
print(f"\n Training failed!")
print(f" Error: {progress['error']}")
break
elif progress['status'] == 'running':
print(f" Epoch {progress['current_epoch']}/{progress['total_epochs']}, Loss: {progress['current_loss']:.6f}", end='\r')
else:
print(f"\n ⚠️ Training still running after 30 seconds")
print(f"\n Training still running after 30 seconds")
progress = training_adapter.get_training_progress(training_id)
print(f" Status: {progress['status']}")
print(f" Epoch: {progress['current_epoch']}/{progress['total_epochs']}")
except Exception as e:
print(f" ❌ Training failed with exception: {e}")
print(f" Training failed with exception: {e}")
import traceback
traceback.print_exc()

100
web/clean_dashboard.py
View File

@@ -216,7 +216,7 @@ class CleanTradingDashboard:
# Connect dashboard to orchestrator for COB data updates
if hasattr(self.orchestrator, 'set_dashboard'):
self.orchestrator.set_dashboard(self)
logger.info("✅ Dashboard connected to orchestrator for COB data updates")
logger.info(" Dashboard connected to orchestrator for COB data updates")
# Start orchestrator's real-time processing to ensure COB data flows
if hasattr(self.orchestrator, 'start_continuous_trading'):
@@ -234,7 +234,7 @@ class CleanTradingDashboard:
trading_thread = threading.Thread(target=start_orchestrator_trading, daemon=True)
trading_thread.start()
logger.info("✅ Started orchestrator real-time processing for COB data")
logger.info(" Started orchestrator real-time processing for COB data")
except Exception as e:
logger.error(f"Failed to start orchestrator trading: {e}")
@@ -835,7 +835,7 @@ class CleanTradingDashboard:
if success:
# Create callbacks for the new model
self._create_model_toggle_callbacks(model_name)
logger.info(f"✅ Successfully added model dynamically: {model_name}")
logger.info(f" Successfully added model dynamically: {model_name}")
return True
else:
logger.error(f"Failed to register model with orchestrator: {model_name}")
@@ -861,7 +861,7 @@ class CleanTradingDashboard:
if hasattr(self.orchestrator, 'model_registry'):
self.orchestrator.model_registry.unregister_model(model_name)
logger.info(f"✅ Successfully removed model dynamically: {model_name}")
logger.info(f" Successfully removed model dynamically: {model_name}")
return True
else:
logger.error("No orchestrator available for dynamic model removal")
@@ -7839,19 +7839,19 @@ class CleanTradingDashboard:
self._force_dashboard_refresh()
logger.info("=" * 60)
logger.info("✅ SESSION CLEAR COMPLETED SUCCESSFULLY")
logger.info(" SESSION CLEAR COMPLETED SUCCESSFULLY")
logger.info("=" * 60)
logger.info("📊 Session P&L reset to $0.00")
logger.info("📈 All positions closed")
logger.info("📋 Trade history cleared")
logger.info("🎯 Success rate calculations reset")
logger.info("📈 Model performance metrics reset")
logger.info("🔄 All caches cleared")
logger.info(" All caches cleared")
logger.info("📁 Trade log files cleared")
logger.info("=" * 60)
except Exception as e:
logger.error(f"❌ Error clearing session: {e}")
logger.error(f" Error clearing session: {e}")
def _close_all_positions(self):
"""Close all held positions"""
@@ -7894,10 +7894,10 @@ class CleanTradingDashboard:
if hasattr(self, 'realized_pnl'):
self.realized_pnl = 0.0
logger.info("✅ All positions closed and PnL reset")
logger.info(" All positions closed and PnL reset")
except Exception as e:
logger.error(f"❌ Error closing positions: {e}")
logger.error(f" Error closing positions: {e}")
def _clear_trade_logs(self):
"""Clear all trade log files"""
@@ -7960,27 +7960,27 @@ class CleanTradingDashboard:
# Use the orchestrator's built-in clear method if available
if hasattr(self.orchestrator, 'clear_session_data'):
self.orchestrator.clear_session_data()
logger.info("✅ Used orchestrator's built-in clear_session_data method")
logger.info(" Used orchestrator's built-in clear_session_data method")
else:
# Fallback to manual clearing
if hasattr(self.orchestrator, 'recent_decisions'):
self.orchestrator.recent_decisions = {}
logger.info("✅ Cleared recent_decisions")
logger.info(" Cleared recent_decisions")
if hasattr(self.orchestrator, 'recent_dqn_predictions'):
for symbol in self.orchestrator.recent_dqn_predictions:
self.orchestrator.recent_dqn_predictions[symbol].clear()
logger.info("✅ Cleared recent_dqn_predictions")
logger.info(" Cleared recent_dqn_predictions")
if hasattr(self.orchestrator, 'recent_cnn_predictions'):
for symbol in self.orchestrator.recent_cnn_predictions:
self.orchestrator.recent_cnn_predictions[symbol].clear()
logger.info("✅ Cleared recent_cnn_predictions")
logger.info(" Cleared recent_cnn_predictions")
if hasattr(self.orchestrator, 'prediction_accuracy_history'):
for symbol in self.orchestrator.prediction_accuracy_history:
self.orchestrator.prediction_accuracy_history[symbol].clear()
logger.info("✅ Cleared prediction_accuracy_history")
logger.info(" Cleared prediction_accuracy_history")
logger.info("Orchestrator state cleared (fallback method)")
@@ -7994,7 +7994,7 @@ class CleanTradingDashboard:
'accuracy': 0.0,
'price_predictions': {'total': 0, 'accurate': 0, 'avg_error': 0.0}
}
logger.info("✅ Reset model_performance tracking (accuracy calculations)")
logger.info(" Reset model_performance tracking (accuracy calculations)")
# Clear model statistics if they exist
if hasattr(self.orchestrator, 'model_statistics'):
@@ -8005,7 +8005,7 @@ class CleanTradingDashboard:
self.orchestrator.model_statistics[model_name].correct = 0
if hasattr(self.orchestrator.model_statistics[model_name], 'total'):
self.orchestrator.model_statistics[model_name].total = 0
logger.info("✅ Reset model_statistics accuracy tracking")
logger.info(" Reset model_statistics accuracy tracking")
# Clear any cached performance metrics
if hasattr(self.orchestrator, '_cached_performance'):
@@ -8014,7 +8014,7 @@ class CleanTradingDashboard:
if hasattr(self.orchestrator, '_last_performance_update'):
self.orchestrator._last_performance_update = {}
logger.info("✅ Orchestrator state and performance metrics cleared completely")
logger.info(" Orchestrator state and performance metrics cleared completely")
except Exception as e:
logger.error(f"Error clearing orchestrator state: {e}")
@@ -8035,11 +8035,11 @@ class CleanTradingDashboard:
# Clear trade history and records (critical for success rate calculations)
if hasattr(self.trading_executor, 'trade_history'):
self.trading_executor.trade_history = []
logger.info("✅ Cleared trade_history")
logger.info(" Cleared trade_history")
if hasattr(self.trading_executor, 'trade_records'):
self.trading_executor.trade_records = []
logger.info("✅ Cleared trade_records (used for success rate)")
logger.info(" Cleared trade_records (used for success rate)")
# Clear P&L and fee tracking
if hasattr(self.trading_executor, 'session_pnl'):
@@ -8060,19 +8060,19 @@ class CleanTradingDashboard:
# Clear consecutive loss tracking (affects success rate calculations)
if hasattr(self.trading_executor, 'consecutive_losses'):
self.trading_executor.consecutive_losses = 0
logger.info("✅ Reset consecutive_losses counter")
logger.info(" Reset consecutive_losses counter")
# Reset safety feature state
if hasattr(self.trading_executor, 'safety_triggered'):
self.trading_executor.safety_triggered = False
logger.info("✅ Reset safety_triggered flag")
logger.info(" Reset safety_triggered flag")
# Reset profitability multiplier to default
if hasattr(self.trading_executor, 'profitability_reward_multiplier'):
self.trading_executor.profitability_reward_multiplier = getattr(
self.trading_executor, 'default_profitability_multiplier', 1.0
)
logger.info("✅ Reset profitability_reward_multiplier")
logger.info(" Reset profitability_reward_multiplier")
# Clear any cached statistics
if hasattr(self.trading_executor, '_cached_stats'):
@@ -8081,7 +8081,7 @@ class CleanTradingDashboard:
if hasattr(self.trading_executor, '_last_stats_update'):
self.trading_executor._last_stats_update = None
logger.info("✅ Trading executor state cleared completely")
logger.info(" Trading executor state cleared completely")
logger.info("📊 Success rate calculations will start fresh")
except Exception as e:
@@ -8127,7 +8127,7 @@ class CleanTradingDashboard:
logger.info("💾 Storing Decision Fusion model...")
# Add storage logic here
except Exception as e:
logger.warning(f"❌ Failed to store Decision Fusion model: {e}")
logger.warning(f" Failed to store Decision Fusion model: {e}")
# 5. Verification Step - Try to load checkpoints to verify they work
logger.info("🔍 Verifying stored checkpoints...")
@@ -8153,14 +8153,14 @@ class CleanTradingDashboard:
if result:
file_path, metadata = result
verification_results.append((model_name, True, f"Verified: {metadata.checkpoint_id}"))
logger.info(f"✅ Verified {model_name} checkpoint: {metadata.checkpoint_id}")
logger.info(f" Verified {model_name} checkpoint: {metadata.checkpoint_id}")
else:
verification_results.append((model_name, False, "Checkpoint not found after save"))
logger.warning(f"⚠️ Could not verify {model_name} checkpoint")
logger.warning(f" Could not verify {model_name} checkpoint")
except Exception as e:
verification_results.append((model_name, False, f"Verification failed: {str(e)}"))
logger.warning(f"⚠️ Failed to verify {model_name}: {e}")
logger.warning(f" Failed to verify {model_name}: {e}")
# 6. Store session metadata
try:
@@ -8203,18 +8203,18 @@ class CleanTradingDashboard:
if stored_models:
logger.info(f"📊 STORAGE SUMMARY:")
logger.info(f" ✅ Models stored: {successful_stores}")
logger.info(f" ✅ Verifications passed: {successful_verifications}/{len(verification_results)}")
logger.info(f" Models stored: {successful_stores}")
logger.info(f" Verifications passed: {successful_verifications}/{len(verification_results)}")
logger.info(f" 📋 Models: {[name for name, _ in stored_models]}")
# Update button display with success info
return True
else:
logger.warning("❌ No models were stored - no models available")
logger.warning(" No models were stored - no models available")
return False
except Exception as e:
logger.error(f"❌ Error in store all models operation: {e}")
logger.error(f" Error in store all models operation: {e}")
import traceback
traceback.print_exc()
return False
@@ -8355,7 +8355,7 @@ class CleanTradingDashboard:
)
if success:
logger.info("✅ Unified storage system enabled for real-time training")
logger.info(" Unified storage system enabled for real-time training")
# Get initial statistics
stats = self.data_provider.get_unified_storage_stats()
@@ -8364,7 +8364,7 @@ class CleanTradingDashboard:
logger.info(f" Database: Connected and ready")
logger.info(f" Ingestion: Pipeline active")
else:
logger.warning("⚠️ Unified storage initialization failed, using legacy data access")
logger.warning(" Unified storage initialization failed, using legacy data access")
except Exception as e:
logger.warning(f"Could not enable unified storage: {e}")
@@ -8955,10 +8955,10 @@ class CleanTradingDashboard:
"""Initialize enhanced COB integration with WebSocket status monitoring"""
try:
if not COB_INTEGRATION_AVAILABLE:
logger.warning("⚠️ COB integration not available - WebSocket status will show as unavailable")
logger.warning(" COB integration not available - WebSocket status will show as unavailable")
return
logger.info("🚀 Initializing Enhanced COB Integration with WebSocket monitoring")
logger.info(" Initializing Enhanced COB Integration with WebSocket monitoring")
# Initialize COB integration
self.cob_integration = COBIntegration(
@@ -8978,15 +8978,15 @@ class CleanTradingDashboard:
loop.run_until_complete(self.cob_integration.start())
loop.run_forever()
except Exception as e:
logger.error(f"❌ Error in COB integration thread: {e}")
logger.error(f" Error in COB integration thread: {e}")
cob_thread = threading.Thread(target=start_cob_integration, daemon=True)
cob_thread.start()
logger.info("✅ Enhanced COB Integration started with WebSocket monitoring")
logger.info(" Enhanced COB Integration started with WebSocket monitoring")
except Exception as e:
logger.error(f"❌ Error initializing Enhanced COB Integration: {e}")
logger.error(f" Error initializing Enhanced COB Integration: {e}")
def update_cob_data_from_orchestrator(self, symbol: str, cob_data: Dict):
"""Update COB cache from orchestrator data - called by orchestrator"""
@@ -9098,7 +9098,7 @@ class CleanTradingDashboard:
logger.debug(f"📊 Enhanced COB update for {symbol}: {websocket_status} via {source}")
except Exception as e:
logger.error(f"❌ Error handling enhanced COB update for {symbol}: {e}")
logger.error(f" Error handling enhanced COB update for {symbol}: {e}")
def get_cob_websocket_status(self) -> Dict[str, Any]:
"""Get COB WebSocket status for dashboard display"""
@@ -9158,13 +9158,13 @@ class CleanTradingDashboard:
status_summary['warning_message'] = None
elif connected_count + fallback_count == total_symbols:
status_summary['overall_status'] = 'partial_fallback'
status_summary['warning_message'] = f'⚠️ {fallback_count} symbol(s) using REST fallback - WebSocket connection failed'
status_summary['warning_message'] = f' {fallback_count} symbol(s) using REST fallback - WebSocket connection failed'
elif fallback_count > 0:
status_summary['overall_status'] = 'degraded'
status_summary['warning_message'] = f'⚠️ COB WebSocket degraded - {error_count} error(s), {fallback_count} fallback(s)'
status_summary['warning_message'] = f' COB WebSocket degraded - {error_count} error(s), {fallback_count} fallback(s)'
else:
status_summary['overall_status'] = 'error'
status_summary['warning_message'] = '❌ COB WebSocket failed - All connections down'
status_summary['warning_message'] = ' COB WebSocket failed - All connections down'
# Set last update time and calculate overall update rate
last_updates = [cache.get('last_update', 0) for cache in self.cob_cache.values()]
@@ -9178,7 +9178,7 @@ class CleanTradingDashboard:
return status_summary
except Exception as e:
logger.error(f"❌ Error getting COB WebSocket status: {e}")
logger.error(f" Error getting COB WebSocket status: {e}")
return {
'overall_status': 'error',
'warning_message': f'Error getting status: {e}',
@@ -9385,7 +9385,7 @@ class CleanTradingDashboard:
'executed': False,
'blocked': False,
'manual': False,
'cob_snapshot': cob_snapshot, # ✅ STORE FULL INFERENCE SNAPSHOT
'cob_snapshot': cob_snapshot, # STORE FULL INFERENCE SNAPSHOT
'inference_data': {
'imbalance': imbalance,
'abs_imbalance': abs_imbalance,
@@ -9439,7 +9439,7 @@ class CleanTradingDashboard:
'history': self.cob_data_history[symbol][-15:], # Last 15 seconds
'bucketed_data': self.cob_bucketed_data[symbol],
'cumulative_imbalance': cumulative_imbalance,
'cob_imbalance_ma': self.cob_imbalance_ma.get(symbol, {}), # ✅ ADD MOVING AVERAGES
'cob_imbalance_ma': self.cob_imbalance_ma.get(symbol, {}), # ADD MOVING AVERAGES
'timestamp': cob_snapshot['timestamp'],
'stats': cob_snapshot.get('stats', {}),
'bids': cob_snapshot.get('bids', []),
@@ -9895,7 +9895,7 @@ class CleanTradingDashboard:
logger.info(f"🔗 Running chained inference for {symbol} with {n_steps} steps")
if self.orchestrator is None:
logger.warning("❌ No orchestrator available for chained inference")
logger.warning(" No orchestrator available for chained inference")
return
# Trigger initial predictions by calling make_trading_decision
@@ -9918,7 +9918,7 @@ class CleanTradingDashboard:
logger.info(f"🔗 Chained inference completed for {symbol}")
except Exception as e:
logger.error(f"❌ Error in chained inference: {e}")
logger.error(f" Error in chained inference: {e}")
# Run the async inference
try:
@@ -9934,7 +9934,7 @@ class CleanTradingDashboard:
asyncio.run(_run_inference())
except Exception as e:
logger.error(f"❌ Chained inference failed: {e}")
logger.error(f" Chained inference failed: {e}")
def run_server(self, host='127.0.0.1', port=8050, debug=False):
"""Start the Dash server"""
@@ -10024,7 +10024,7 @@ class CleanTradingDashboard:
try:
if self.orchestrator and hasattr(self.orchestrator, 'add_decision_callback'):
self.orchestrator.add_decision_callback(self._on_trading_decision)
logger.info("✅ Orchestrator decision callback registered")
logger.info(" Orchestrator decision callback registered")
else:
logger.warning("Orchestrator not available or doesn't support callbacks")
except Exception as e:
@@ -10205,7 +10205,7 @@ class CleanTradingDashboard:
self.decision_fusion_inference_enabled = toggle_states['decision_fusion'].get('inference_enabled', True)
self.decision_fusion_training_enabled = toggle_states['decision_fusion'].get('training_enabled', True)
logger.info(f"✅ UI state synced from orchestrator: DQN(inf:{self.dqn_inference_enabled}, train:{self.dqn_training_enabled}), CNN(inf:{self.cnn_inference_enabled}, train:{self.cnn_training_enabled}), COB_RL(inf:{getattr(self, 'cob_rl_inference_enabled', True)}, train:{getattr(self, 'cob_rl_training_enabled', True)}), Decision_Fusion(inf:{getattr(self, 'decision_fusion_inference_enabled', True)}, train:{getattr(self, 'decision_fusion_training_enabled', True)})")
logger.info(f" UI state synced from orchestrator: DQN(inf:{self.dqn_inference_enabled}, train:{self.dqn_training_enabled}), CNN(inf:{self.cnn_inference_enabled}, train:{self.cnn_training_enabled}), COB_RL(inf:{getattr(self, 'cob_rl_inference_enabled', True)}, train:{getattr(self, 'cob_rl_training_enabled', True)}), Decision_Fusion(inf:{getattr(self, 'decision_fusion_inference_enabled', True)}, train:{getattr(self, 'decision_fusion_training_enabled', True)})")
else:
logger.debug("Orchestrator not available for UI state sync, using defaults")
except Exception as e:

26
web/cob_dashboard.html
View File

@@ -553,7 +553,7 @@
console.log(`🔌 Unhandled WebSocket message type:`, data.type);
}
} catch (error) {
console.error('❌ Error parsing WebSocket message:', error);
console.error(' Error parsing WebSocket message:', error);
console.error('Raw message:', event.data);
}
};
@@ -584,7 +584,7 @@
}
// Debug logging to understand data structure
console.log(`🔄 ${symbol} COB Update:`, {
console.log(` ${symbol} COB Update:`, {
source: data.type || 'Unknown',
bidsCount: (cobData.bids || []).length,
asksCount: (cobData.asks || []).length,
@@ -602,7 +602,7 @@
const asks = cobData.asks || [];
if (bids.length <= 1 && asks.length <= 1) {
console.log(`⚠️ Insufficient WS depth for ${symbol}, fetching REST data...`);
console.log(` Insufficient WS depth for ${symbol}, fetching REST data...`);
fetchRESTData(symbol);
return;
}
@@ -630,7 +630,7 @@
drawMiniChart(prefix, cobData.ohlcv);
}, 100);
} else {
console.log(`❌ ${symbol}: No valid OHLCV data in update (${cobData.ohlcv ? cobData.ohlcv.length : 'null'} items)`);
console.log(` ${symbol}: No valid OHLCV data in update (${cobData.ohlcv ? cobData.ohlcv.length : 'null'} items)`);
// Try to get OHLCV from REST endpoint
console.log(`🔍 Trying to fetch OHLCV from REST for ${symbol}...`);
@@ -671,13 +671,13 @@
});
if (data.data) {
console.log(`✅ Processing REST fallback data for ${symbol}`);
console.log(` Processing REST fallback data for ${symbol}`);
handleCOBUpdate({symbol: symbol, data: data.data, type: 'rest_api'});
} else {
console.error(`❌ No data in REST response for ${symbol}`);
console.error(` No data in REST response for ${symbol}`);
}
})
.catch(error => console.error(`❌ Error fetching REST data for ${symbol}:`, error));
.catch(error => console.error(` Error fetching REST data for ${symbol}:`, error));
}
function trackImbalance(symbol, imbalance) {
@@ -1070,7 +1070,7 @@
try {
const canvas = document.getElementById(`${prefix}-mini-chart`);
if (!canvas) {
console.error(`❌ Canvas not found for ${prefix}-mini-chart`);
console.error(` Canvas not found for ${prefix}-mini-chart`);
return;
}
@@ -1090,14 +1090,14 @@
ctx.font = '12px Courier New';
ctx.textAlign = 'center';
ctx.fillText('No Data', width / 2, height / 2);
console.log(`❌ ${prefix}: No OHLCV data to draw`);
console.log(` ${prefix}: No OHLCV data to draw`);
return;
}
// Validate OHLCV data structure
const firstCandle = ohlcvArray[0];
if (!firstCandle || typeof firstCandle.open === 'undefined' || typeof firstCandle.close === 'undefined') {
console.error(`❌ ${prefix}: Invalid OHLCV data structure:`, firstCandle);
console.error(` ${prefix}: Invalid OHLCV data structure:`, firstCandle);
ctx.fillStyle = '#ff6b6b';
ctx.font = '10px Courier New';
ctx.textAlign = 'center';
@@ -1118,7 +1118,7 @@
console.log(`📊 ${prefix} price range: $${minPrice.toFixed(2)} - $${maxPrice.toFixed(2)} (range: $${priceRange.toFixed(2)})`);
if (priceRange === 0) {
console.warn(`⚠️ ${prefix}: Zero price range, cannot draw chart`);
console.warn(` ${prefix}: Zero price range, cannot draw chart`);
ctx.fillStyle = '#ff6b6b';
ctx.font = '10px Courier New';
ctx.textAlign = 'center';
@@ -1187,10 +1187,10 @@
ctx.setLineDash([]);
}
console.log(`✅ Successfully drew ${prefix} chart with ${ohlcvArray.length} candles`);
console.log(` Successfully drew ${prefix} chart with ${ohlcvArray.length} candles`);
} catch (error) {
console.error(`❌ Error drawing mini chart for ${prefix}:`, error);
console.error(` Error drawing mini chart for ${prefix}:`, error);
console.error(error.stack);
}
}