264 lines
7.0 KiB
Markdown
264 lines
7.0 KiB
Markdown
# COBY Portainer Deployment Guide
|
|
|
|
This guide explains how to deploy the COBY Multi-Exchange Data Aggregation System using Portainer with Git repository integration.
|
|
|
|
## Prerequisites
|
|
|
|
- Portainer CE/EE installed and running
|
|
- Docker Swarm or Docker Compose environment
|
|
- Access to the Git repository containing the COBY project
|
|
- Minimum system requirements:
|
|
- 4GB RAM
|
|
- 2 CPU cores
|
|
- 20GB disk space
|
|
|
|
## Deployment Steps
|
|
|
|
### 1. Access Portainer
|
|
|
|
1. Open your Portainer web interface
|
|
2. Navigate to your environment (local Docker or Docker Swarm)
|
|
|
|
### 2. Create Stack from Git Repository
|
|
|
|
1. Go to **Stacks** in the left sidebar
|
|
2. Click **Add stack**
|
|
3. Choose **Repository** as the build method
|
|
4. Configure the repository settings:
|
|
|
|
**Repository Configuration:**
|
|
- **Repository URL**: `https://github.com/your-username/your-repo.git`
|
|
- **Repository reference**: `main` (or your preferred branch)
|
|
- **Compose path**: `COBY/docker-compose.portainer.yml`
|
|
- **Additional files**: Leave empty (all configs are embedded)
|
|
|
|
### 3. Configure Environment Variables
|
|
|
|
In the **Environment variables** section, add the following variables (optional customizations):
|
|
|
|
```bash
|
|
# Database Configuration
|
|
DB_PASSWORD=your_secure_database_password
|
|
REDIS_PASSWORD=your_secure_redis_password
|
|
|
|
# API Configuration
|
|
API_PORT=8080
|
|
WS_PORT=8081
|
|
|
|
# Monitoring (if using monitoring profile)
|
|
PROMETHEUS_PORT=9090
|
|
GRAFANA_PORT=3001
|
|
GRAFANA_PASSWORD=your_grafana_password
|
|
|
|
# Performance Tuning
|
|
MAX_CONNECTIONS_PER_EXCHANGE=5
|
|
DATA_BUFFER_SIZE=10000
|
|
BATCH_WRITE_SIZE=1000
|
|
```
|
|
|
|
### 4. Deploy the Stack
|
|
|
|
1. **Stack name**: Enter `coby-system` (or your preferred name)
|
|
2. **Environment variables**: Configure as needed (see above)
|
|
3. **Access control**: Set appropriate permissions
|
|
4. Click **Deploy the stack**
|
|
|
|
### 5. Monitor Deployment
|
|
|
|
1. Watch the deployment logs in Portainer
|
|
2. Check that all services start successfully:
|
|
- `coby-timescaledb` (Database)
|
|
- `coby-redis` (Cache)
|
|
- `coby-app` (Main application)
|
|
- `coby-dashboard` (Web interface)
|
|
|
|
### 6. Verify Installation
|
|
|
|
Once deployed, verify the installation:
|
|
|
|
1. **Health Checks**: All services should show as "healthy" in Portainer
|
|
2. **Web Dashboard**: Access `http://your-server:8080/` (served by your reverse proxy)
|
|
3. **API Endpoint**: Check `http://your-server:8080/health`
|
|
4. **Logs**: Review logs for any errors
|
|
|
|
**Reverse Proxy Configuration**: Configure your reverse proxy to forward requests to the COBY app on port 8080. The application serves both the API and web dashboard from the same port.
|
|
|
|
## Service Ports
|
|
|
|
The following ports will be exposed:
|
|
|
|
- **8080**: REST API + Web Dashboard (served by FastAPI)
|
|
- **8081**: WebSocket API
|
|
- **5432**: TimescaleDB (optional external access)
|
|
- **6379**: Redis (optional external access)
|
|
|
|
**Note**: The web dashboard is now served directly by the FastAPI application at port 8080, eliminating the need for a separate nginx container since you have a reverse proxy.
|
|
|
|
## Optional Monitoring Stack
|
|
|
|
To enable Prometheus and Grafana monitoring:
|
|
|
|
1. In the stack configuration, add the profile: `monitoring`
|
|
2. Additional ports will be exposed:
|
|
- **9090**: Prometheus
|
|
- **3001**: Grafana
|
|
- **9100**: Node Exporter
|
|
|
|
## Configuration Options
|
|
|
|
### Resource Limits
|
|
|
|
The stack includes resource limits for each service:
|
|
|
|
- **COBY App**: 2GB RAM, 2 CPU cores (includes web dashboard)
|
|
- **TimescaleDB**: 1GB RAM, 1 CPU core
|
|
- **Redis**: 512MB RAM, 0.5 CPU cores
|
|
|
|
### Persistent Data
|
|
|
|
The following volumes are created for persistent data:
|
|
|
|
- `timescale_data`: Database storage
|
|
- `redis_data`: Redis persistence
|
|
- `coby_logs`: Application logs
|
|
- `coby_data`: Application data
|
|
- `prometheus_data`: Metrics storage (if monitoring enabled)
|
|
- `grafana_data`: Grafana dashboards (if monitoring enabled)
|
|
|
|
### Network Configuration
|
|
|
|
- **Network**: `coby-network` (172.20.0.0/16)
|
|
- **Internal communication**: All services communicate via Docker network
|
|
- **External access**: Only specified ports are exposed
|
|
|
|
## Troubleshooting
|
|
|
|
### Common Issues
|
|
|
|
1. **Services not starting**:
|
|
- Check resource availability
|
|
- Review service logs in Portainer
|
|
- Verify environment variables
|
|
|
|
2. **Database connection issues**:
|
|
- Ensure TimescaleDB is healthy
|
|
- Check database credentials
|
|
- Verify network connectivity
|
|
|
|
3. **Web dashboard not accessible**:
|
|
- Confirm port 8080 is accessible through your reverse proxy
|
|
- Check that coby-app is running and healthy
|
|
- Verify static files are being served at the root path
|
|
|
|
### Log Access
|
|
|
|
Access logs through Portainer:
|
|
|
|
1. Go to **Containers**
|
|
2. Click on the container name
|
|
3. Select **Logs** tab
|
|
4. Use filters to find specific issues
|
|
|
|
### Health Checks
|
|
|
|
Monitor service health:
|
|
|
|
1. **Portainer Dashboard**: Shows health status
|
|
2. **API Health**: `GET /health` endpoint
|
|
3. **Database**: `pg_isready` command
|
|
4. **Redis**: `redis-cli ping` command
|
|
|
|
## Scaling and Updates
|
|
|
|
### Horizontal Scaling
|
|
|
|
To scale the main application:
|
|
|
|
1. Go to the stack in Portainer
|
|
2. Edit the stack
|
|
3. Modify the `coby-app` service replicas
|
|
4. Redeploy the stack
|
|
|
|
### Updates
|
|
|
|
To update the system:
|
|
|
|
1. **Git-based updates**: Portainer will pull latest changes
|
|
2. **Manual updates**: Edit stack configuration
|
|
3. **Rolling updates**: Use Docker Swarm mode for zero-downtime updates
|
|
|
|
### Backup
|
|
|
|
Regular backups should include:
|
|
|
|
- **Database**: TimescaleDB data volume
|
|
- **Configuration**: Stack configuration in Portainer
|
|
- **Logs**: Application logs for troubleshooting
|
|
|
|
## Security Considerations
|
|
|
|
1. **Change default passwords** for database and Redis
|
|
2. **Use environment variables** for sensitive data
|
|
3. **Limit network exposure** to required ports only
|
|
4. **Regular updates** of base images
|
|
5. **Monitor logs** for security events
|
|
|
|
## Performance Tuning
|
|
|
|
### Database Optimization
|
|
|
|
- Adjust `shared_buffers` in TimescaleDB
|
|
- Configure connection pooling
|
|
- Monitor query performance
|
|
|
|
### Application Tuning
|
|
|
|
- Adjust `DATA_BUFFER_SIZE` for throughput
|
|
- Configure `BATCH_WRITE_SIZE` for database writes
|
|
- Monitor memory usage and adjust limits
|
|
|
|
### Network Optimization
|
|
|
|
- Use Docker overlay networks for multi-host deployments
|
|
- Configure load balancing for high availability
|
|
- Monitor network latency between services
|
|
|
|
## Support
|
|
|
|
For issues and support:
|
|
|
|
1. Check the application logs
|
|
2. Review Portainer container status
|
|
3. Consult the main project documentation
|
|
4. Submit issues to the project repository
|
|
|
|
## Example Stack Configuration
|
|
|
|
Here's a complete example of environment variables for production:
|
|
|
|
```bash
|
|
# Production Configuration
|
|
ENVIRONMENT=production
|
|
DEBUG=false
|
|
LOG_LEVEL=INFO
|
|
|
|
# Security
|
|
DB_PASSWORD=prod_secure_db_pass_2024
|
|
REDIS_PASSWORD=prod_secure_redis_pass_2024
|
|
|
|
# Performance
|
|
MAX_CONNECTIONS_PER_EXCHANGE=10
|
|
DATA_BUFFER_SIZE=20000
|
|
BATCH_WRITE_SIZE=2000
|
|
|
|
# Monitoring
|
|
PROMETHEUS_PORT=9090
|
|
GRAFANA_PORT=3001
|
|
GRAFANA_PASSWORD=secure_grafana_pass
|
|
|
|
# Exchange Configuration
|
|
EXCHANGES=binance,coinbase,kraken,bybit,okx,huobi,kucoin,gateio,bitfinex,mexc
|
|
SYMBOLS=BTCUSDT,ETHUSDT,ADAUSDT,DOTUSDT
|
|
```
|
|
|
|
This configuration provides a robust production deployment suitable for high-throughput cryptocurrency data aggregation. |