gogo2/ARCHITECTURE_REFACTORING_PLAN.md

Architecture Refactoring Plan

Current Issues

1. Duplicate Core Implementations

• ANNOTATE/core/data_loader.py vs core/data_provider.py - overlapping data loading
• ANNOTATE/core/inference_training_system.py vs core/orchestrator.py - overlapping training coordination
• ANNOTATE/core/real_training_adapter.py - should be in main core
• Multiple data models scattered across both cores

2. Import Dependencies

• Main core imports from ANNOTATE/core (wrong direction)
• Circular dependencies between systems
• Inconsistent data flow

3. Responsibilities Overlap

• Both orchestrator and InferenceTrainingCoordinator handle training
• Both data_provider and data_loader handle data fetching
• Duplicate model management

Refactoring Strategy

Phase 1: Move Core Classes to Main Core

1.1 Move InferenceFrameReference to core/data_models.py

# Move from: ANNOTATE/core/inference_training_system.py
# To: core/data_models.py
@dataclass
class InferenceFrameReference:
    # ... existing implementation

1.2 Integrate InferenceTrainingCoordinator into Orchestrator

# In core/orchestrator.py - merge functionality instead of importing
class TradingOrchestrator:
    def __init__(self):
        # Integrate training coordination directly
        self.training_event_subscribers = []
        self.inference_frames = {}
        # ... merge InferenceTrainingCoordinator methods

1.3 Move RealTrainingAdapter to Main Core

# Move from: ANNOTATE/core/real_training_adapter.py  
# To: core/enhanced_rl_training_adapter.py (extend existing)

Phase 2: Eliminate ANNOTATE/core/data_loader.py

2.1 Extend Main DataProvider

# In core/data_provider.py - add methods from HistoricalDataLoader
class DataProvider:
    def get_data_for_annotation(self, symbol, timeframe, start_time=None, end_time=None, limit=2500, direction='latest'):
        """Method specifically for annotation UI needs"""
        # Implement annotation-specific data loading
        
    def get_multi_timeframe_data(self, symbol, timeframes, start_time=None, end_time=None, limit=2500):
        """Multi-timeframe data for annotation UI"""
        # Implement multi-timeframe loading

2.2 Update ANNOTATE App

# In ANNOTATE/web/app.py
from core.data_provider import DataProvider  # Use main data provider directly

class AnnotationDashboard:
    def __init__(self):
        # Use main data provider instead of wrapper
        self.data_provider = DataProvider(config)

Phase 3: Consolidate Training Systems

3.1 Merge Training Responsibilities

# In core/orchestrator.py
class TradingOrchestrator:
    def subscribe_training_events(self, callback, event_types):
        """Unified training event subscription"""
        
    def store_inference_frame(self, symbol, timeframe, prediction_data):
        """Store inference frames for training"""
        
    def trigger_training_on_event(self, event_type, event_data):
        """Unified training trigger system"""

3.2 Remove Duplicate Classes

• Delete ANNOTATE/core/inference_training_system.py
• Delete ANNOTATE/core/data_loader.py
• Move useful methods to main core classes

Phase 4: Clean Architecture

4.1 Single Data Flow

Exchange APIs → DataProvider → Orchestrator → Models
                     ↓              ↓
              ANNOTATE UI ← Training System

4.2 Clear Responsibilities

• core/data_provider.py: All data fetching, caching, real-time integration
• core/orchestrator.py: All model coordination, training events, inference
• core/data_models.py: All shared data structures
• ANNOTATE/: UI only, no core logic

Implementation Steps

Step 1: Move InferenceFrameReference

1. Copy class to core/data_models.py
2. Update imports in orchestrator
3. Remove from ANNOTATE/core/

Step 2: Integrate Training Coordination

1. Move InferenceTrainingCoordinator methods into orchestrator
2. Update ANNOTATE app to use orchestrator directly
3. Remove duplicate training system

Step 3: Extend DataProvider

1. Add annotation-specific methods to main DataProvider
2. Update ANNOTATE app to use main DataProvider
3. Remove ANNOTATE/core/data_loader.py

Step 4: Clean Up

1. Remove ANNOTATE/core/ directory entirely
2. Update all imports
3. Test live data flow

Expected Benefits

1. Single Source of Truth

• One DataProvider handling all data
• One Orchestrator handling all training
• One set of data models

2. Proper Live Data Flow

• WebSocket → DataProvider → API → Charts
• No duplicate caching or stale data

3. Cleaner Architecture

• ANNOTATE becomes pure UI
• Core contains all business logic
• Clear dependency direction

4. Easier Maintenance

• No duplicate code to maintain
• Single place to fix issues
• Consistent behavior across apps

Files to Modify

Move/Merge:

• ANNOTATE/core/inference_training_system.py → core/orchestrator.py
• ANNOTATE/core/real_training_adapter.py → core/enhanced_rl_training_adapter.py
• InferenceFrameReference → core/data_models.py

Update:

• ANNOTATE/web/app.py (use main core classes)
• core/orchestrator.py (integrate training coordination)
• core/data_provider.py (add annotation methods)

Delete:

• ANNOTATE/core/data_loader.py
• ANNOTATE/core/inference_training_system.py (after merge)
• Entire ANNOTATE/core/ directory (eventually)