gogo2/COBY/docker/README.md

Market Data Infrastructure Docker Setup

This directory contains Docker Compose configurations and scripts for deploying TimescaleDB and Redis infrastructure for the multi-exchange data aggregation system.

🏗️ Architecture

• TimescaleDB: Time-series database optimized for high-frequency market data
• Redis: High-performance caching layer for real-time data
• Network: Isolated Docker network for secure communication

📋 Prerequisites

• Docker Engine 20.10+
• Docker Compose 2.0+
• At least 4GB RAM available for containers
• 50GB+ disk space for data storage

🚀 Quick Start

1. Copy environment file:

   cp .env.example .env
   

2. Edit configuration (update passwords and settings):

   nano .env
   

3. Deploy infrastructure:

   chmod +x deploy.sh
   ./deploy.sh
   

4. Verify deployment:

   docker-compose -f timescaledb-compose.yml ps
   

📁 File Structure

docker/
├── timescaledb-compose.yml    # Main Docker Compose configuration
├── init-scripts/              # Database initialization scripts
│   └── 01-init-timescaledb.sql
├── redis.conf                 # Redis configuration
├── .env                       # Environment variables
├── deploy.sh                  # Deployment script
├── backup.sh                  # Backup script
├── restore.sh                 # Restore script
└── README.md                  # This file

⚙️ Configuration

Environment Variables

Key variables in .env:

# Database credentials
POSTGRES_PASSWORD=your_secure_password
POSTGRES_USER=market_user
POSTGRES_DB=market_data

# Redis settings
REDIS_PASSWORD=your_redis_password

# Performance tuning
POSTGRES_SHARED_BUFFERS=256MB
POSTGRES_EFFECTIVE_CACHE_SIZE=1GB
REDIS_MAXMEMORY=2gb

TimescaleDB Configuration

The database is pre-configured with:

• Optimized PostgreSQL settings for time-series data
• TimescaleDB extension enabled
• Hypertables for automatic partitioning
• Retention policies (90 days for raw data)
• Continuous aggregates for common queries
• Proper indexes for query performance

Redis Configuration

Redis is configured for:

• High-frequency data caching
• Memory optimization (2GB limit)
• Persistence with AOF and RDB
• Optimized for order book data structures

🔌 Connection Details

After deployment, connect using:

TimescaleDB

Host: 192.168.0.10
Port: 5432
Database: market_data
Username: market_user
Password: (from .env file)

Redis

Host: 192.168.0.10
Port: 6379
Password: (from .env file)

🗄️ Database Schema

The system creates the following tables:

• order_book_snapshots: Real-time order book data
• trade_events: Individual trade events
• heatmap_data: Aggregated price bucket data
• ohlcv_data: OHLCV candlestick data
• exchange_status: Exchange connection monitoring
• system_metrics: System performance metrics

💾 Backup & Restore

Create Backup

chmod +x backup.sh
./backup.sh

Backups are stored in ./backups/ with timestamp.

Restore from Backup

chmod +x restore.sh
./restore.sh market_data_backup_YYYYMMDD_HHMMSS.tar.gz

Automated Backups

Set up a cron job for regular backups:

# Daily backup at 2 AM
0 2 * * * /path/to/docker/backup.sh

📊 Monitoring

Health Checks

Check service health:

# TimescaleDB
docker exec market_data_timescaledb pg_isready -U market_user -d market_data

# Redis
docker exec market_data_redis redis-cli -a your_password ping

View Logs

# All services
docker-compose -f timescaledb-compose.yml logs -f

# Specific service
docker-compose -f timescaledb-compose.yml logs -f timescaledb

Database Queries

Connect to TimescaleDB:

docker exec -it market_data_timescaledb psql -U market_user -d market_data

Example queries:

-- Check table sizes
SELECT 
    schemaname,
    tablename,
    pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) as size
FROM pg_tables 
WHERE schemaname = 'market_data';

-- Recent order book data
SELECT * FROM market_data.order_book_snapshots 
ORDER BY timestamp DESC LIMIT 10;

-- Exchange status
SELECT * FROM market_data.exchange_status 
ORDER BY timestamp DESC LIMIT 10;

🔧 Maintenance

Update Images

docker-compose -f timescaledb-compose.yml pull
docker-compose -f timescaledb-compose.yml up -d

Clean Up Old Data

# TimescaleDB has automatic retention policies
# Manual cleanup if needed:
docker exec market_data_timescaledb psql -U market_user -d market_data -c "
SELECT drop_chunks('market_data.order_book_snapshots', INTERVAL '30 days');
"

Scale Resources

Edit timescaledb-compose.yml to adjust:

• Memory limits
• CPU limits
• Shared buffers
• Connection limits

🚨 Troubleshooting

Common Issues

1. Port conflicts: Change ports in compose file if 5432/6379 are in use
2. Memory issues: Reduce shared_buffers and Redis maxmemory
3. Disk space: Monitor /var/lib/docker/volumes/ usage
4. Connection refused: Check firewall settings and container status

Performance Tuning

1. TimescaleDB:

   • Adjust shared_buffers based on available RAM
   • Tune effective_cache_size to 75% of system RAM
   • Monitor query performance with pg_stat_statements

2. Redis:

   • Adjust maxmemory based on data volume
   • Monitor memory usage with INFO memory
   • Use appropriate eviction policy

Recovery Procedures

1. Container failure: docker-compose restart <service>
2. Data corruption: Restore from latest backup
3. Network issues: Check Docker network configuration
4. Performance degradation: Review logs and system metrics

🔐 Security

• Change default passwords in .env
• Use strong passwords (20+ characters)
• Restrict network access to trusted IPs
• Regular security updates
• Monitor access logs
• Enable SSL/TLS for production

📞 Support

For issues related to:

• TimescaleDB: Check TimescaleDB docs
• Redis: Check Redis docs
• Docker: Check Docker docs

🔄 Updates

This infrastructure supports:

• Rolling updates with zero downtime
• Blue-green deployments
• Automated failover
• Data migration scripts