gogo2/.kiro/specs/multi-exchange-data-aggregation/design.md

Design Document

Overview

The Multi-Exchange Data Aggregation System is a comprehensive data collection and processing subsystem designed to serve as the foundational data layer for the trading orchestrator. The system will collect real-time order book and OHLCV data from the top 10 cryptocurrency exchanges, aggregate it into standardized formats, store it in a TimescaleDB time-series database, and provide both live data feeds and historical replay capabilities.

The system follows a microservices architecture with containerized components, ensuring scalability, maintainability, and seamless integration with the existing trading infrastructure.

We implement it in the .\COBY subfolder for easy integration with the existing system

Architecture

High-Level Architecture

graph TB
    subgraph "Exchange Connectors"
        E1[Binance WebSocket]
        E2[Coinbase WebSocket]
        E3[Kraken WebSocket]
        E4[Bybit WebSocket]
        E5[OKX WebSocket]
        E6[Huobi WebSocket]
        E7[KuCoin WebSocket]
        E8[Gate.io WebSocket]
        E9[Bitfinex WebSocket]
        E10[MEXC WebSocket]
    end
    
    subgraph "Data Processing Layer"
        DP[Data Processor]
        AGG[Aggregation Engine]
        NORM[Data Normalizer]
    end
    
    subgraph "Storage Layer"
        TSDB[(TimescaleDB)]
        CACHE[Redis Cache]
    end
    
    subgraph "API Layer"
        LIVE[Live Data API]
        REPLAY[Replay API]
        WEB[Web Dashboard]
    end
    
    subgraph "Integration Layer"
        ORCH[Orchestrator Interface]
        ADAPTER[Data Adapter]
    end
    
    E1 --> DP
    E2 --> DP
    E3 --> DP
    E4 --> DP
    E5 --> DP
    E6 --> DP
    E7 --> DP
    E8 --> DP
    E9 --> DP
    E10 --> DP
    
    DP --> NORM
    NORM --> AGG
    AGG --> TSDB
    AGG --> CACHE
    
    CACHE --> LIVE
    TSDB --> REPLAY
    LIVE --> WEB
    REPLAY --> WEB
    
    LIVE --> ADAPTER
    REPLAY --> ADAPTER
    ADAPTER --> ORCH

Component Architecture

The system is organized into several key components:

1. Exchange Connectors: WebSocket clients for each exchange
2. Data Processing Engine: Normalizes and validates incoming data
3. Aggregation Engine: Creates price buckets and heatmaps
4. Storage Layer: TimescaleDB for persistence, Redis for caching
5. API Layer: REST and WebSocket APIs for data access
6. Web Dashboard: Real-time visualization interface
7. Integration Layer: Orchestrator-compatible interface

Components and Interfaces

Exchange Connector Interface

class ExchangeConnector:
    """Base interface for exchange WebSocket connectors"""
    
    async def connect(self) -> bool
    async def disconnect(self) -> None
    async def subscribe_orderbook(self, symbol: str) -> None
    async def subscribe_trades(self, symbol: str) -> None
    def get_connection_status(self) -> ConnectionStatus
    def add_data_callback(self, callback: Callable) -> None

Data Processing Interface

class DataProcessor:
    """Processes and normalizes raw exchange data"""
    
    def normalize_orderbook(self, raw_data: Dict, exchange: str) -> OrderBookSnapshot
    def normalize_trade(self, raw_data: Dict, exchange: str) -> TradeEvent
    def validate_data(self, data: Union[OrderBookSnapshot, TradeEvent]) -> bool
    def calculate_metrics(self, orderbook: OrderBookSnapshot) -> OrderBookMetrics

Aggregation Engine Interface

class AggregationEngine:
    """Aggregates data into price buckets and heatmaps"""
    
    def create_price_buckets(self, orderbook: OrderBookSnapshot, bucket_size: float) -> PriceBuckets
    def update_heatmap(self, symbol: str, buckets: PriceBuckets) -> HeatmapData
    def calculate_imbalances(self, orderbook: OrderBookSnapshot) -> ImbalanceMetrics
    def aggregate_across_exchanges(self, symbol: str) -> ConsolidatedOrderBook

Storage Interface

class StorageManager:
    """Manages data persistence and retrieval"""
    
    async def store_orderbook(self, data: OrderBookSnapshot) -> bool
    async def store_trade(self, data: TradeEvent) -> bool
    async def get_historical_data(self, symbol: str, start: datetime, end: datetime) -> List[Dict]
    async def get_latest_data(self, symbol: str) -> Dict
    def setup_database_schema(self) -> None

Replay Interface

class ReplayManager:
    """Provides historical data replay functionality"""
    
    def create_replay_session(self, start_time: datetime, end_time: datetime, speed: float) -> str
    async def start_replay(self, session_id: str) -> None
    async def pause_replay(self, session_id: str) -> None
    async def stop_replay(self, session_id: str) -> None
    def get_replay_status(self, session_id: str) -> ReplayStatus

Data Models

Core Data Structures

@dataclass
class OrderBookSnapshot:
    """Standardized order book snapshot"""
    symbol: str
    exchange: str
    timestamp: datetime
    bids: List[PriceLevel]
    asks: List[PriceLevel]
    sequence_id: Optional[int] = None
    
@dataclass
class PriceLevel:
    """Individual price level in order book"""
    price: float
    size: float
    count: Optional[int] = None

@dataclass
class TradeEvent:
    """Standardized trade event"""
    symbol: str
    exchange: str
    timestamp: datetime
    price: float
    size: float
    side: str  # 'buy' or 'sell'
    trade_id: str

@dataclass
class PriceBuckets:
    """Aggregated price buckets for heatmap"""
    symbol: str
    timestamp: datetime
    bucket_size: float
    bid_buckets: Dict[float, float]  # price -> volume
    ask_buckets: Dict[float, float]  # price -> volume
    
@dataclass
class HeatmapData:
    """Heatmap visualization data"""
    symbol: str
    timestamp: datetime
    bucket_size: float
    data: List[HeatmapPoint]
    
@dataclass
class HeatmapPoint:
    """Individual heatmap data point"""
    price: float
    volume: float
    intensity: float  # 0.0 to 1.0
    side: str  # 'bid' or 'ask'

Database Schema

TimescaleDB Tables

-- Order book snapshots table
CREATE TABLE order_book_snapshots (
    id BIGSERIAL,
    symbol VARCHAR(20) NOT NULL,
    exchange VARCHAR(20) NOT NULL,
    timestamp TIMESTAMPTZ NOT NULL,
    bids JSONB NOT NULL,
    asks JSONB NOT NULL,
    sequence_id BIGINT,
    mid_price DECIMAL(20,8),
    spread DECIMAL(20,8),
    bid_volume DECIMAL(30,8),
    ask_volume DECIMAL(30,8),
    PRIMARY KEY (timestamp, symbol, exchange)
);

-- Convert to hypertable
SELECT create_hypertable('order_book_snapshots', 'timestamp');

-- Trade events table
CREATE TABLE trade_events (
    id BIGSERIAL,
    symbol VARCHAR(20) NOT NULL,
    exchange VARCHAR(20) NOT NULL,
    timestamp TIMESTAMPTZ NOT NULL,
    price DECIMAL(20,8) NOT NULL,
    size DECIMAL(30,8) NOT NULL,
    side VARCHAR(4) NOT NULL,
    trade_id VARCHAR(100) NOT NULL,
    PRIMARY KEY (timestamp, symbol, exchange, trade_id)
);

-- Convert to hypertable
SELECT create_hypertable('trade_events', 'timestamp');

-- Aggregated heatmap data table
CREATE TABLE heatmap_data (
    symbol VARCHAR(20) NOT NULL,
    timestamp TIMESTAMPTZ NOT NULL,
    bucket_size DECIMAL(10,2) NOT NULL,
    price_bucket DECIMAL(20,8) NOT NULL,
    volume DECIMAL(30,8) NOT NULL,
    side VARCHAR(3) NOT NULL,
    exchange_count INTEGER NOT NULL,
    PRIMARY KEY (timestamp, symbol, bucket_size, price_bucket, side)
);

-- Convert to hypertable
SELECT create_hypertable('heatmap_data', 'timestamp');

-- OHLCV data table
CREATE TABLE ohlcv_data (
    symbol VARCHAR(20) NOT NULL,
    timestamp TIMESTAMPTZ NOT NULL,
    timeframe VARCHAR(10) NOT NULL,
    open_price DECIMAL(20,8) NOT NULL,
    high_price DECIMAL(20,8) NOT NULL,
    low_price DECIMAL(20,8) NOT NULL,
    close_price DECIMAL(20,8) NOT NULL,
    volume DECIMAL(30,8) NOT NULL,
    trade_count INTEGER,
    PRIMARY KEY (timestamp, symbol, timeframe)
);

-- Convert to hypertable
SELECT create_hypertable('ohlcv_data', 'timestamp');

Error Handling

Connection Management

The system implements robust error handling for exchange connections:

1. Exponential Backoff: Failed connections retry with increasing delays
2. Circuit Breaker: Temporarily disable problematic exchanges
3. Graceful Degradation: Continue operation with available exchanges
4. Health Monitoring: Continuous monitoring of connection status

Data Validation

All incoming data undergoes validation:

1. Schema Validation: Ensure data structure compliance
2. Range Validation: Check price and volume ranges
3. Timestamp Validation: Verify temporal consistency
4. Duplicate Detection: Prevent duplicate data storage

Database Resilience

Database operations include comprehensive error handling:

1. Connection Pooling: Maintain multiple database connections
2. Transaction Management: Ensure data consistency
3. Retry Logic: Automatic retry for transient failures
4. Backup Strategies: Regular data backups and recovery procedures

Testing Strategy

Unit Testing

Each component will have comprehensive unit tests:

1. Exchange Connectors: Mock WebSocket responses
2. Data Processing: Test normalization and validation
3. Aggregation Engine: Verify bucket calculations
4. Storage Layer: Test database operations
5. API Layer: Test endpoint responses

Integration Testing

End-to-end testing scenarios:

1. Multi-Exchange Data Flow: Test complete data pipeline
2. Database Integration: Verify TimescaleDB operations
3. API Integration: Test orchestrator interface compatibility
4. Performance Testing: Load testing with high-frequency data

Performance Testing

Performance benchmarks and testing:

1. Throughput Testing: Measure data processing capacity
2. Latency Testing: Measure end-to-end data latency
3. Memory Usage: Monitor memory consumption patterns
4. Database Performance: Query performance optimization

Monitoring and Observability

Comprehensive monitoring system:

1. Metrics Collection: Prometheus-compatible metrics
2. Logging: Structured logging with correlation IDs
3. Alerting: Real-time alerts for system issues
4. Dashboards: Grafana dashboards for system monitoring

Deployment Architecture

Docker Containerization

The system will be deployed using Docker containers:

# docker-compose.yml
version: '3.8'
services:
  timescaledb:
    image: timescale/timescaledb:latest-pg14
    environment:
      POSTGRES_DB: market_data
      POSTGRES_USER: market_user
      POSTGRES_PASSWORD: ${DB_PASSWORD}
    volumes:
      - timescale_data:/var/lib/postgresql/data
    ports:
      - "5432:5432"
    
  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    volumes:
      - redis_data:/data
    
  data-aggregator:
    build: ./data-aggregator
    environment:
      - DB_HOST=timescaledb
      - REDIS_HOST=redis
      - LOG_LEVEL=INFO
    depends_on:
      - timescaledb
      - redis
    
  web-dashboard:
    build: ./web-dashboard
    ports:
      - "8080:8080"
    environment:
      - API_HOST=data-aggregator
    depends_on:
      - data-aggregator

volumes:
  timescale_data:
  redis_data:

Configuration Management

Environment-based configuration:

# config.py
@dataclass
class Config:
    # Database settings
    db_host: str = os.getenv('DB_HOST', 'localhost')
    db_port: int = int(os.getenv('DB_PORT', '5432'))
    db_name: str = os.getenv('DB_NAME', 'market_data')
    db_user: str = os.getenv('DB_USER', 'market_user')
    db_password: str = os.getenv('DB_PASSWORD', '')
    
    # Redis settings
    redis_host: str = os.getenv('REDIS_HOST', 'localhost')
    redis_port: int = int(os.getenv('REDIS_PORT', '6379'))
    
    # Exchange settings
    exchanges: List[str] = field(default_factory=lambda: [
        'binance', 'coinbase', 'kraken', 'bybit', 'okx',
        'huobi', 'kucoin', 'gateio', 'bitfinex', 'mexc'
    ])
    
    # Aggregation settings
    btc_bucket_size: float = 10.0  # $10 USD buckets for BTC
    eth_bucket_size: float = 1.0   # $1 USD buckets for ETH
    
    # Performance settings
    max_connections_per_exchange: int = 5
    data_buffer_size: int = 10000
    batch_write_size: int = 1000
    
    # API settings
    api_host: str = os.getenv('API_HOST', '0.0.0.0')
    api_port: int = int(os.getenv('API_PORT', '8080'))
    websocket_port: int = int(os.getenv('WS_PORT', '8081'))

This design provides a robust, scalable foundation for multi-exchange data aggregation that seamlessly integrates with the existing trading orchestrator while providing the flexibility for future enhancements and additional exchange integrations.