kiro steering, live training wip

2025-11-13 15:09:20 +02:00
parent 1af3124be7
commit 25287d0e9e
8 changed files with 1319 additions and 302 deletions
--- a/.kiro/steering/tech.md
+++ b/.kiro/steering/tech.md
@@ -0,0 +1,181 @@
+# Technology Stack
+
+## Core Technologies
+
+### Python Ecosystem
+- **Python 3.x**: Primary language
+- **PyTorch**: Deep learning framework (CPU/CUDA/DirectML support)
+- **NumPy/Pandas**: Data manipulation and analysis
+- **scikit-learn**: ML utilities and preprocessing
+
+### Web & API
+- **Dash/Plotly**: Interactive web dashboard
+- **Flask**: ANNOTATE web UI
+- **FastAPI**: COBY REST API
+- **WebSockets**: Real-time data streaming
+
+### Data Storage
+- **DuckDB**: Primary data storage (time-series optimized)
+- **SQLite**: Metadata and predictions database
+- **Redis**: High-performance caching (COBY)
+- **TimescaleDB**: Optional time-series storage (COBY)
+
+### Exchange Integration
+- **ccxt**: Multi-exchange API library
+- **websocket-client**: Real-time market data
+- **pybit**: Bybit-specific integration
+
+### Monitoring & Logging
+- **TensorBoard**: Training visualization
+- **wandb**: Experiment tracking
+- **structlog**: Structured logging (COBY)
+
+## Hardware Acceleration
+
+### GPU Support
+- NVIDIA CUDA (via PyTorch CUDA builds)
+- AMD DirectML (via onnxruntime-directml)
+- CPU fallback (default PyTorch CPU build)
+
+**Note**: PyTorch is NOT in requirements.txt to avoid pulling NVIDIA CUDA deps on AMD machines. Install manually based on hardware.
+
+## Project Structure
+
+```
+gogo2/
+├── core/              # Core trading system components
+├── models/            # Trained model checkpoints
+├── NN/                # Neural network models and training
+├── COBY/              # Multi-exchange data aggregation
+├── ANNOTATE/          # Manual annotation UI
+├── web/               # Main dashboard
+├── utils/             # Shared utilities
+├── cache/             # Data caching
+├── data/              # Databases and exports
+├── logs/              # System logs
+└── @checkpoints/      # Model checkpoints archive
+```
+
+## Configuration
+
+- **config.yaml**: Main system configuration (exchanges, symbols, timeframes, trading params)
+- **models.yml**: Model-specific settings (CNN, RL, training)
+- **.env**: Sensitive credentials (API keys, database passwords)
+
+## Common Commands
+
+### Running the System
+
+```bash
+# Main dashboard with live training
+python main_dashboard.py --port 8051
+
+# Dashboard without training
+python main_dashboard.py --port 8051 --no-training
+
+# Clean dashboard (alternative)
+python run_clean_dashboard.py
+```
+
+### Training
+
+```bash
+# Unified training runner - realtime mode
+python training_runner.py --mode realtime --duration 4
+
+# Backtest training
+python training_runner.py --mode backtest --start-date 2024-01-01 --end-date 2024-12-31
+
+# CNN training with TensorBoard
+python main_clean.py --mode cnn --symbol ETH/USDT
+tensorboard --logdir=runs
+
+# RL training
+python main_clean.py --mode rl --symbol ETH/USDT
+```
+
+### Backtesting
+
+```bash
+# 30-day backtest
+python main_backtest.py --start 2024-01-01 --end 2024-01-31
+
+# Custom symbol and window
+python main_backtest.py --start 2024-01-01 --end 2024-12-31 --symbol BTC/USDT --window 48
+```
+
+### COBY System
+
+```bash
+# Start COBY data aggregation
+python COBY/main.py --debug
+
+# Access COBY dashboard: http://localhost:8080
+# COBY API: http://localhost:8080/api/...
+# COBY WebSocket: ws://localhost:8081/dashboard
+```
+
+### ANNOTATE System
+
+```bash
+# Start annotation UI
+python ANNOTATE/web/app.py
+
+# Access at: http://127.0.0.1:8051
+```
+
+### Testing
+
+```bash
+# Run tests
+python -m pytest tests/
+
+# Test specific components
+python test_cnn_only.py
+python test_training.py
+python test_duckdb_storage.py
+```
+
+### Monitoring
+
+```bash
+# TensorBoard for training metrics
+tensorboard --logdir=runs
+# Access at: http://localhost:6006
+
+# Check data stream status
+python check_stream.py status
+python check_stream.py ohlcv
+python check_stream.py cob
+```
+
+## Development Tools
+
+- **TensorBoard**: Training visualization (runs/ directory)
+- **wandb**: Experiment tracking
+- **pytest**: Testing framework
+- **Git**: Version control
+
+## Dependencies Management
+
+```bash
+# Install dependencies
+pip install -r requirements.txt
+
+# Install PyTorch (choose based on hardware)
+# CPU-only:
+pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
+
+# NVIDIA GPU (CUDA 12.1):
+pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
+
+# AMD NPU:
+pip install onnxruntime-directml onnx transformers optimum
+```
+
+## Performance Targets
+
+- **Memory Usage**: <2GB per model, <28GB total system
+- **Training Speed**: ~20 seconds for 50 epochs
+- **Inference Latency**: <200ms per prediction
+- **Real Data Processing**: 1000+ candles per timeframe