gogo2/.kiro/steering/structure.md

Project Structure & Architecture

Module Organization

core/ - Core Trading System

Central trading logic and data management.

Key modules:

• orchestrator.py: Decision coordination, combines CNN/RL predictions
• data_provider.py: Real market data fetching (Binance API)
• data_models.py: Shared data structures (OHLCV, features, predictions)
• config.py: Configuration management
• trading_executor.py: Order execution and position management
• exchanges/: Exchange-specific implementations (Binance, Bybit, Deribit, MEXC)

Multi-horizon system:

• multi_horizon_prediction_manager.py: Generates 1m/5m/15m/60m predictions
• multi_horizon_trainer.py: Deferred training when outcomes known
• prediction_snapshot_storage.py: Efficient prediction storage

Training:

• extrema_trainer.py: Trains on market extrema (pivots)
• training_integration.py: Training pipeline integration
• overnight_training_coordinator.py: Scheduled training sessions

NN/ - Neural Network Models

Deep learning models for pattern recognition and trading decisions.

models/:

• enhanced_cnn.py: CNN for pattern recognition (100M params)
• standardized_cnn.py: Standardized CNN interface
• advanced_transformer_trading.py: Transformer for long-range dependencies
• dqn_agent.py: Deep Q-Network for RL trading
• model_interfaces.py: Abstract interfaces for all models

training/:

• Training pipelines for each model type
• Batch processing and optimization

utils/:

• data_interface.py: Connects to realtime data
• Feature engineering and preprocessing

COBY/ - Data Aggregation System

Multi-exchange order book and OHLCV data collection.

Structure:

• main.py: Entry point
• config.py: COBY-specific configuration
• models/core.py: Data models (OrderBookSnapshot, TradeEvent, PriceBuckets)
• interfaces/: Abstract interfaces for connectors, processors, storage
• api/rest_api.py: FastAPI REST endpoints
• web/static/: Dashboard UI (http://localhost:8080)
• connectors/: Exchange WebSocket connectors
• storage/: TimescaleDB/Redis integration
• monitoring/: System monitoring and metrics

ANNOTATE/ - Manual Annotation UI

Web interface for marking profitable trades on historical data.

Structure:

• web/app.py: Flask/Dash application
• web/templates/: Jinja2 HTML templates
• core/annotation_manager.py: Annotation storage and retrieval
• core/training_simulator.py: Simulates training with annotations
• core/data_loader.py: Historical data loading
• data/annotations/: Saved annotations
• data/test_cases/: Generated training test cases

web/ - Main Dashboard

Real-time monitoring and visualization.

Key files:

• clean_dashboard.py: Main dashboard application
• cob_realtime_dashboard.py: COB-specific dashboard
• component_manager.py: UI component management
• layout_manager.py: Dashboard layout
• models_training_panel.py: Training controls
• prediction_chart.py: Prediction visualization

models/ - Model Checkpoints

Trained model weights and checkpoints.

Organization:

• cnn/: CNN model checkpoints
• rl/: RL model checkpoints
• enhanced_cnn/: Enhanced CNN variants
• enhanced_rl/: Enhanced RL variants
• best_models/: Best performing models
• checkpoints/: Training checkpoints

utils/ - Shared Utilities

Common functionality across modules.

Key utilities:

• checkpoint_manager.py: Model checkpoint save/load
• cache_manager.py: Data caching
• database_manager.py: SQLite database operations
• inference_logger.py: Prediction logging
• timezone_utils.py: Timezone handling
• training_integration.py: Training pipeline utilities

data/ - Data Storage

Databases and cached data.

Contents:

• predictions.db: SQLite prediction database
• trading_system.db: Trading metadata
• cache/: Cached market data
• prediction_snapshots/: Stored predictions for training
• text_exports/: Exported data for analysis

cache/ - Data Caching

High-performance data caching.

Contents:

• trading_data.duckdb: DuckDB time-series storage
• parquet_store/: Parquet files for efficient storage
• monthly_1s_data/: Monthly 1-second data cache
• pivot_bounds/: Cached pivot calculations

@checkpoints/ - Checkpoint Archive

Archived model checkpoints organized by type.

Organization:

• cnn/, dqn/, hybrid/, rl/, transformer/: By model type
• best_models/: Best performers
• archive/: Historical checkpoints

Architecture Patterns

Data Flow

Exchange APIs → DataProvider → Orchestrator → Models (CNN/RL/Transformer)
                                    ↓
                            Trading Executor → Exchange APIs

Training Flow

Real Market Data → Feature Engineering → Model Training → Checkpoint Save
                                              ↓
                                    Validation & Metrics

Multi-Horizon Flow

Orchestrator → PredictionManager → Generate predictions (1m/5m/15m/60m)
                                          ↓
                                  SnapshotStorage
                                          ↓
                            Wait for target time (deferred)
                                          ↓
                            MultiHorizonTrainer → Train models

COBY Data Flow

Exchange WebSockets → Connectors → DataProcessor → AggregationEngine
                                                          ↓
                                                  StorageManager
                                                          ↓
                                            TimescaleDB + Redis

Dependency Patterns

Core Dependencies

• orchestrator.py depends on: all models, data_provider, trading_executor
• data_provider.py depends on: cache_manager, timezone_utils
• Models depend on: data_models, checkpoint_manager

Dashboard Dependencies

• clean_dashboard.py depends on: orchestrator, data_provider, all models
• Uses component_manager and layout_manager for UI

Circular Dependency Prevention

• Use abstract interfaces (model_interfaces.py)
• Dependency injection for orchestrator
• Lazy imports where needed

Configuration Hierarchy

1. config.yaml: Main system config (exchanges, symbols, trading params)
2. models.yml: Model-specific settings (architecture, training)
3. .env: Sensitive credentials (API keys, passwords)
4. Module-specific configs in each subsystem (COBY/config.py, etc.)

Naming Conventions

Files

• Snake_case for Python files: data_provider.py
• Descriptive names: multi_horizon_prediction_manager.py

Classes

• PascalCase: DataProvider, MultiHorizonTrainer
• Descriptive: PredictionSnapshotStorage

Functions

• Snake_case: get_ohlcv_data(), train_model()
• Verb-noun pattern: calculate_features(), save_checkpoint()

Variables

• Snake_case: prediction_data, model_output
• Descriptive: cnn_confidence_threshold

Import Patterns

Absolute imports preferred

from core.data_provider import DataProvider
from NN.models.enhanced_cnn import EnhancedCNN

Relative imports for same package

from .data_models import OHLCV
from ..utils import checkpoint_manager

Testing Structure

• Unit tests in tests/ directory
• Integration tests: test_integration.py
• Component-specific tests: test_cnn_only.py, test_training.py
• Use pytest framework

Documentation

• Module-level docstrings in each file
• README.md in major subsystems (COBY/, NN/, ANNOTATE/)
• Architecture docs in root: COB_MODEL_ARCHITECTURE_DOCUMENTATION.md, MULTI_HORIZON_TRAINING_SYSTEM.md
• Implementation summaries: IMPLEMENTATION_SUMMARY.md, TRAINING_IMPROVEMENTS_SUMMARY.md