kiro steering, live training wip

2025-11-13 15:09:20 +02:00
parent 1af3124be7
commit 25287d0e9e
8 changed files with 1319 additions and 302 deletions
--- a/ANNOTATE/LIVE_PIVOT_TRAINING_GUIDE.md
+++ b/ANNOTATE/LIVE_PIVOT_TRAINING_GUIDE.md
@@ -0,0 +1,332 @@
+# Live Pivot Training - Automatic Training on Market Structure
+
+## Overview
+
+The Live Pivot Training system automatically trains your models on significant market structure points (L2 pivots) detected in real-time on 1s and 1m charts.
+
+**Key Benefits:**
+- ✅ Continuous learning from live market data
+- ✅ Trains on high-quality pivot points (peaks and troughs)
+- ✅ Non-blocking - doesn't interfere with inference
+- ✅ Automatic - no manual annotation needed
+- ✅ Adaptive - learns from current market conditions
+
+## How It Works
+
+### 1. Pivot Detection
+```
+Live Market Data (1s, 1m)
+    ↓
+Williams Market Structure
+    ↓
+L2 Pivot Detection
+    ↓
+High/Low Identification
+```
+
+### 2. Training Sample Creation
+When an L2 pivot is detected:
+- **High Pivot** → Creates SHORT training sample
+- **Low Pivot** → Creates LONG training sample
+
+### 3. Background Training
+- Training happens in separate thread
+- Doesn't block inference
+- Uses same training pipeline as manual annotations
+
+## Usage
+
+### Starting Live Inference with Auto-Training
+
+**Default (Auto-training ENABLED):**
+```javascript
+fetch('/api/realtime-inference/start', {
+    method: 'POST',
+    headers: { 'Content-Type': 'application/json' },
+    body: JSON.stringify({
+        model_name: 'Transformer',
+        symbol: 'ETH/USDT'
+        // enable_live_training: true (default)
+    })
+})
+```
+
+**Disable Auto-Training:**
+```javascript
+fetch('/api/realtime-inference/start', {
+    method: 'POST',
+    headers: { 'Content-Type': 'application/json' },
+    body: JSON.stringify({
+        model_name: 'Transformer',
+        symbol: 'ETH/USDT',
+        enable_live_training: false  // Disable
+    })
+})
+```
+
+### Python API
+
+```python
+from ANNOTATE.core.live_pivot_trainer import get_live_pivot_trainer
+
+# Get trainer instance
+pivot_trainer = get_live_pivot_trainer(
+    orchestrator=orchestrator,
+    data_provider=data_provider,
+    training_adapter=training_adapter
+)
+
+# Start monitoring
+pivot_trainer.start(symbol='ETH/USDT')
+
+# Get statistics
+stats = pivot_trainer.get_stats()
+print(f"Trained on {stats['total_trained_pivots']} pivots")
+
+# Stop monitoring
+pivot_trainer.stop()
+```
+
+## Configuration
+
+### Adjustable Parameters
+
+Located in `ANNOTATE/core/live_pivot_trainer.py`:
+
+```python
+class LivePivotTrainer:
+    def __init__(self, ...):
+        # Check for new pivots every 5 seconds
+        self.check_interval = 5
+        
+        # Minimum 60 seconds between training on same timeframe
+        self.min_pivot_spacing = 60
+        
+        # Track last 1000 trained pivots (avoid duplicates)
+        self.trained_pivots = deque(maxlen=1000)
+```
+
+### Timeframe Configuration
+
+Currently monitors:
+- **1s timeframe** - High-frequency pivots
+- **1m timeframe** - Short-term pivots
+
+Can be extended to 1h, 1d by modifying `_monitoring_loop()`.
+
+## Training Sample Structure
+
+Each L2 pivot generates a training sample:
+
+```python
+{
+    'test_case_id': 'live_pivot_ETH/USDT_1m_2025-11-12T18:30:00',
+    'symbol': 'ETH/USDT',
+    'timestamp': '2025-11-12T18:30:00+00:00',
+    'action': 'BUY',  # or 'SELL' for high pivots
+    'expected_outcome': {
+        'direction': 'LONG',  # or 'SHORT'
+        'entry_price': 3150.50,
+        'exit_price': None,  # Determined by model
+        'profit_loss_pct': 0.0,
+        'holding_period_seconds': 300  # 5 minutes default
+    },
+    'training_config': {
+        'timeframes': ['1s', '1m', '1h', '1d'],
+        'candles_per_timeframe': 200
+    },
+    'annotation_metadata': {
+        'source': 'live_pivot_detection',
+        'pivot_level': 'L2',
+        'pivot_type': 'low',  # or 'high'
+        'confidence': 0.85
+    }
+}
+```
+
+## Monitoring
+
+### Log Messages
+
+**Startup:**
+```
+LivePivotTrainer initialized
+LivePivotTrainer started for ETH/USDT
+✅ Live pivot training ENABLED - will train on L2 peaks automatically
+```
+
+**Pivot Detection:**
+```
+Found 2 new L2 pivots on ETH/USDT 1m
+Training on L2 low pivot @ 3150.50 on ETH/USDT 1m
+Started background training on L2 pivot
+Live pivot training session started: abc-123-def
+```
+
+**Statistics:**
+```python
+stats = pivot_trainer.get_stats()
+# {
+#     'running': True,
+#     'total_trained_pivots': 47,
+#     'last_training_1s': 1699876543.21,
+#     'last_training_1m': 1699876540.15,
+#     'pivot_history_1s': 100,
+#     'pivot_history_1m': 100
+# }
+```
+
+## Performance Considerations
+
+### Memory Usage
+- Tracks last 1000 trained pivots (~50KB)
+- Pivot history: 100 per timeframe (~10KB)
+- **Total overhead: <100KB**
+
+### CPU Usage
+- Checks every 5 seconds (configurable)
+- Pivot detection: ~10ms per check
+- **Minimal impact on inference**
+
+### Training Frequency
+- Rate limited: 60 seconds between training on same timeframe
+- Prevents overtraining on noisy pivots
+- Typical: 2-10 training sessions per hour
+
+## Best Practices
+
+### 1. Start with Default Settings
+```python
+# Let it run with defaults first
+pivot_trainer.start(symbol='ETH/USDT')
+```
+
+### 2. Monitor Training Quality
+```python
+# Check how many pivots are being trained on
+stats = pivot_trainer.get_stats()
+if stats['total_trained_pivots'] > 100:
+    # Increase min_pivot_spacing to reduce frequency
+    pivot_trainer.min_pivot_spacing = 120  # 2 minutes
+```
+
+### 3. Adjust for Market Conditions
+```python
+# Volatile market - train more frequently
+pivot_trainer.min_pivot_spacing = 30  # 30 seconds
+
+# Quiet market - train less frequently
+pivot_trainer.min_pivot_spacing = 300  # 5 minutes
+```
+
+### 4. Combine with Manual Annotations
+- Live pivot training handles routine patterns
+- Manual annotations for special cases
+- Best of both worlds!
+
+## Troubleshooting
+
+### No Pivots Detected
+**Problem:** `total_trained_pivots` stays at 0
+
+**Solutions:**
+1. Check if Williams Market Structure is initialized:
+   ```python
+   if pivot_trainer.williams_1s is None:
+       # Reinstall/fix Williams Market Structure
+   ```
+
+2. Verify data is flowing:
+   ```python
+   candles = data_provider.get_historical_data('ETH/USDT', '1m', 200)
+   print(f"Candles: {len(candles)}")
+   ```
+
+3. Lower pivot detection threshold (if available)
+
+### Too Many Training Sessions
+**Problem:** Training every few seconds, slowing down system
+
+**Solution:**
+```python
+# Increase spacing
+pivot_trainer.min_pivot_spacing = 180  # 3 minutes
+
+# Or reduce check frequency
+pivot_trainer.check_interval = 10  # Check every 10 seconds
+```
+
+### Training Errors
+**Problem:** Background training fails
+
+**Check logs:**
+```
+Error in background training: ...
+```
+
+**Solutions:**
+1. Verify training adapter is working:
+   ```python
+   # Test manual training first
+   training_adapter.start_training('Transformer', [test_case])
+   ```
+
+2. Check memory availability (training needs RAM)
+
+3. Verify model is loaded in orchestrator
+
+## Integration with Existing Systems
+
+### Works With:
+- ✅ Manual annotation training
+- ✅ Real-time inference
+- ✅ All model types (Transformer, CNN, DQN)
+- ✅ Multiple symbols (start separate instances)
+
+### Doesn't Interfere With:
+- ✅ Live inference predictions
+- ✅ Manual training sessions
+- ✅ Checkpoint saving/loading
+- ✅ Dashboard updates
+
+## Future Enhancements
+
+### Planned Features:
+1. **Adaptive Spacing** - Adjust `min_pivot_spacing` based on market volatility
+2. **Multi-Level Training** - Train on L1, L2, L3 pivots with different priorities
+3. **Outcome Tracking** - Track actual profit/loss of pivot-based trades
+4. **Quality Filtering** - Only train on high-confidence pivots
+5. **Multi-Symbol** - Monitor multiple symbols simultaneously
+
+### Configuration UI (Future):
+```
+┌─────────────────────────────────────┐
+│ Live Pivot Training Settings       │
+├─────────────────────────────────────┤
+│ ☑ Enable Auto-Training              │
+│                                     │
+│ Timeframes:                         │
+│ ☑ 1s  ☑ 1m  ☐ 1h  ☐ 1d            │
+│                                     │
+│ Min Spacing: [60] seconds          │
+│ Check Interval: [5] seconds        │
+│                                     │
+│ Pivot Levels:                       │
+│ ☐ L1  ☑ L2  ☐ L3                   │
+│                                     │
+│ Stats:                              │
+│ Trained: 47 pivots                 │
+│ Last 1s: 2 min ago                 │
+│ Last 1m: 5 min ago                 │
+└─────────────────────────────────────┘
+```
+
+## Summary
+
+Live Pivot Training provides **automatic, continuous learning** from live market data by:
+1. Detecting significant L2 pivot points
+2. Creating training samples automatically
+3. Training models in background
+4. Adapting to current market conditions
+
+**Result:** Your models continuously improve without manual intervention!