popov/gogo2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
71442f766c6664226401fa21e4fd96fc8fdf5f97
gogo2/COBY/PORTAINER_DEPLOYMENT.md
Dobromir Popov 71442f766c 17. docker deployment
2025-08-05 01:22:27 +03:00

7.0 KiB
Raw Blame History

COBY Portainer Deployment Guide

This guide explains how to deploy the COBY Multi-Exchange Data Aggregation System using Portainer with Git repository integration.

Prerequisites

  • Portainer CE/EE installed and running
  • Docker Swarm or Docker Compose environment
  • Access to the Git repository containing the COBY project
  • Minimum system requirements:
    • 4GB RAM
    • 2 CPU cores
    • 20GB disk space

Deployment Steps

1. Access Portainer

  1. Open your Portainer web interface
  2. Navigate to your environment (local Docker or Docker Swarm)

2. Create Stack from Git Repository

  1. Go to Stacks in the left sidebar

  2. Click Add stack

  3. Choose Repository as the build method

  4. Configure the repository settings:

    Repository Configuration:

    • Repository URL: https://github.com/your-username/your-repo.git
    • Repository reference: main (or your preferred branch)
    • Compose path: COBY/docker-compose.portainer.yml
    • Additional files: Leave empty (all configs are embedded)

3. Configure Environment Variables

In the Environment variables section, add the following variables (optional customizations):

# Database Configuration
DB_PASSWORD=your_secure_database_password
REDIS_PASSWORD=your_secure_redis_password

# API Configuration  
API_PORT=8080
WS_PORT=8081

# Monitoring (if using monitoring profile)
PROMETHEUS_PORT=9090
GRAFANA_PORT=3001
GRAFANA_PASSWORD=your_grafana_password

# Performance Tuning
MAX_CONNECTIONS_PER_EXCHANGE=5
DATA_BUFFER_SIZE=10000
BATCH_WRITE_SIZE=1000

4. Deploy the Stack

  1. Stack name: Enter coby-system (or your preferred name)
  2. Environment variables: Configure as needed (see above)
  3. Access control: Set appropriate permissions
  4. Click Deploy the stack

5. Monitor Deployment

  1. Watch the deployment logs in Portainer
  2. Check that all services start successfully:
    • coby-timescaledb (Database)
    • coby-redis (Cache)
    • coby-app (Main application)
    • coby-dashboard (Web interface)

6. Verify Installation

Once deployed, verify the installation:

  1. Health Checks: All services should show as "healthy" in Portainer
  2. Web Dashboard: Access http://your-server:8080/ (served by your reverse proxy)
  3. API Endpoint: Check http://your-server:8080/health
  4. Logs: Review logs for any errors

Reverse Proxy Configuration: Configure your reverse proxy to forward requests to the COBY app on port 8080. The application serves both the API and web dashboard from the same port.

Service Ports

The following ports will be exposed:

  • 8080: REST API + Web Dashboard (served by FastAPI)
  • 8081: WebSocket API
  • 5432: TimescaleDB (optional external access)
  • 6379: Redis (optional external access)

Note: The web dashboard is now served directly by the FastAPI application at port 8080, eliminating the need for a separate nginx container since you have a reverse proxy.

Optional Monitoring Stack

To enable Prometheus and Grafana monitoring:

  1. In the stack configuration, add the profile: monitoring
  2. Additional ports will be exposed:
    • 9090: Prometheus
    • 3001: Grafana
    • 9100: Node Exporter

Configuration Options

Resource Limits

The stack includes resource limits for each service:

  • COBY App: 2GB RAM, 2 CPU cores (includes web dashboard)
  • TimescaleDB: 1GB RAM, 1 CPU core
  • Redis: 512MB RAM, 0.5 CPU cores

Persistent Data

The following volumes are created for persistent data:

  • timescale_data: Database storage
  • redis_data: Redis persistence
  • coby_logs: Application logs
  • coby_data: Application data
  • prometheus_data: Metrics storage (if monitoring enabled)
  • grafana_data: Grafana dashboards (if monitoring enabled)

Network Configuration

  • Network: coby-network (172.20.0.0/16)
  • Internal communication: All services communicate via Docker network
  • External access: Only specified ports are exposed

Troubleshooting

Common Issues

  1. Services not starting:

    • Check resource availability
    • Review service logs in Portainer
    • Verify environment variables

  2. Database connection issues:

    • Ensure TimescaleDB is healthy
    • Check database credentials
    • Verify network connectivity

  3. Web dashboard not accessible:

    • Confirm port 8080 is accessible through your reverse proxy
    • Check that coby-app is running and healthy
    • Verify static files are being served at the root path

Log Access

Access logs through Portainer:

  1. Go to Containers
  2. Click on the container name
  3. Select Logs tab
  4. Use filters to find specific issues

Health Checks

Monitor service health:

  1. Portainer Dashboard: Shows health status
  2. API Health: GET /health endpoint
  3. Database: pg_isready command
  4. Redis: redis-cli ping command

Scaling and Updates

Horizontal Scaling

To scale the main application:

  1. Go to the stack in Portainer
  2. Edit the stack
  3. Modify the coby-app service replicas
  4. Redeploy the stack

Updates

To update the system:

  1. Git-based updates: Portainer will pull latest changes
  2. Manual updates: Edit stack configuration
  3. Rolling updates: Use Docker Swarm mode for zero-downtime updates

Backup

Regular backups should include:

  • Database: TimescaleDB data volume
  • Configuration: Stack configuration in Portainer
  • Logs: Application logs for troubleshooting

Security Considerations

  1. Change default passwords for database and Redis
  2. Use environment variables for sensitive data
  3. Limit network exposure to required ports only
  4. Regular updates of base images
  5. Monitor logs for security events

Performance Tuning

Database Optimization

  • Adjust shared_buffers in TimescaleDB
  • Configure connection pooling
  • Monitor query performance

Application Tuning

  • Adjust DATA_BUFFER_SIZE for throughput
  • Configure BATCH_WRITE_SIZE for database writes
  • Monitor memory usage and adjust limits

Network Optimization

  • Use Docker overlay networks for multi-host deployments
  • Configure load balancing for high availability
  • Monitor network latency between services

Support

For issues and support:

  1. Check the application logs
  2. Review Portainer container status
  3. Consult the main project documentation
  4. Submit issues to the project repository

Example Stack Configuration

Here's a complete example of environment variables for production:

# Production Configuration
ENVIRONMENT=production
DEBUG=false
LOG_LEVEL=INFO

# Security
DB_PASSWORD=prod_secure_db_pass_2024
REDIS_PASSWORD=prod_secure_redis_pass_2024

# Performance
MAX_CONNECTIONS_PER_EXCHANGE=10
DATA_BUFFER_SIZE=20000
BATCH_WRITE_SIZE=2000

# Monitoring
PROMETHEUS_PORT=9090
GRAFANA_PORT=3001
GRAFANA_PASSWORD=secure_grafana_pass

# Exchange Configuration
EXCHANGES=binance,coinbase,kraken,bybit,okx,huobi,kucoin,gateio,bitfinex,mexc
SYMBOLS=BTCUSDT,ETHUSDT,ADAUSDT,DOTUSDT

This configuration provides a robust production deployment suitable for high-throughput cryptocurrency data aggregation.