333 lines
8.7 KiB
Markdown
333 lines
8.7 KiB
Markdown
# Live Pivot Training - Automatic Training on Market Structure
|
|
|
|
## Overview
|
|
|
|
The Live Pivot Training system automatically trains your models on significant market structure points (L2 pivots) detected in real-time on 1s and 1m charts.
|
|
|
|
**Key Benefits:**
|
|
- ✅ Continuous learning from live market data
|
|
- ✅ Trains on high-quality pivot points (peaks and troughs)
|
|
- ✅ Non-blocking - doesn't interfere with inference
|
|
- ✅ Automatic - no manual annotation needed
|
|
- ✅ Adaptive - learns from current market conditions
|
|
|
|
## How It Works
|
|
|
|
### 1. Pivot Detection
|
|
```
|
|
Live Market Data (1s, 1m)
|
|
↓
|
|
Williams Market Structure
|
|
↓
|
|
L2 Pivot Detection
|
|
↓
|
|
High/Low Identification
|
|
```
|
|
|
|
### 2. Training Sample Creation
|
|
When an L2 pivot is detected:
|
|
- **High Pivot** → Creates SHORT training sample
|
|
- **Low Pivot** → Creates LONG training sample
|
|
|
|
### 3. Background Training
|
|
- Training happens in separate thread
|
|
- Doesn't block inference
|
|
- Uses same training pipeline as manual annotations
|
|
|
|
## Usage
|
|
|
|
### Starting Live Inference with Auto-Training
|
|
|
|
**Default (Auto-training ENABLED):**
|
|
```javascript
|
|
fetch('/api/realtime-inference/start', {
|
|
method: 'POST',
|
|
headers: { 'Content-Type': 'application/json' },
|
|
body: JSON.stringify({
|
|
model_name: 'Transformer',
|
|
symbol: 'ETH/USDT'
|
|
// enable_live_training: true (default)
|
|
})
|
|
})
|
|
```
|
|
|
|
**Disable Auto-Training:**
|
|
```javascript
|
|
fetch('/api/realtime-inference/start', {
|
|
method: 'POST',
|
|
headers: { 'Content-Type': 'application/json' },
|
|
body: JSON.stringify({
|
|
model_name: 'Transformer',
|
|
symbol: 'ETH/USDT',
|
|
enable_live_training: false // Disable
|
|
})
|
|
})
|
|
```
|
|
|
|
### Python API
|
|
|
|
```python
|
|
from ANNOTATE.core.live_pivot_trainer import get_live_pivot_trainer
|
|
|
|
# Get trainer instance
|
|
pivot_trainer = get_live_pivot_trainer(
|
|
orchestrator=orchestrator,
|
|
data_provider=data_provider,
|
|
training_adapter=training_adapter
|
|
)
|
|
|
|
# Start monitoring
|
|
pivot_trainer.start(symbol='ETH/USDT')
|
|
|
|
# Get statistics
|
|
stats = pivot_trainer.get_stats()
|
|
print(f"Trained on {stats['total_trained_pivots']} pivots")
|
|
|
|
# Stop monitoring
|
|
pivot_trainer.stop()
|
|
```
|
|
|
|
## Configuration
|
|
|
|
### Adjustable Parameters
|
|
|
|
Located in `ANNOTATE/core/live_pivot_trainer.py`:
|
|
|
|
```python
|
|
class LivePivotTrainer:
|
|
def __init__(self, ...):
|
|
# Check for new pivots every 5 seconds
|
|
self.check_interval = 5
|
|
|
|
# Minimum 60 seconds between training on same timeframe
|
|
self.min_pivot_spacing = 60
|
|
|
|
# Track last 1000 trained pivots (avoid duplicates)
|
|
self.trained_pivots = deque(maxlen=1000)
|
|
```
|
|
|
|
### Timeframe Configuration
|
|
|
|
Currently monitors:
|
|
- **1s timeframe** - High-frequency pivots
|
|
- **1m timeframe** - Short-term pivots
|
|
|
|
Can be extended to 1h, 1d by modifying `_monitoring_loop()`.
|
|
|
|
## Training Sample Structure
|
|
|
|
Each L2 pivot generates a training sample:
|
|
|
|
```python
|
|
{
|
|
'test_case_id': 'live_pivot_ETH/USDT_1m_2025-11-12T18:30:00',
|
|
'symbol': 'ETH/USDT',
|
|
'timestamp': '2025-11-12T18:30:00+00:00',
|
|
'action': 'BUY', # or 'SELL' for high pivots
|
|
'expected_outcome': {
|
|
'direction': 'LONG', # or 'SHORT'
|
|
'entry_price': 3150.50,
|
|
'exit_price': None, # Determined by model
|
|
'profit_loss_pct': 0.0,
|
|
'holding_period_seconds': 300 # 5 minutes default
|
|
},
|
|
'training_config': {
|
|
'timeframes': ['1s', '1m', '1h', '1d'],
|
|
'candles_per_timeframe': 200
|
|
},
|
|
'annotation_metadata': {
|
|
'source': 'live_pivot_detection',
|
|
'pivot_level': 'L2',
|
|
'pivot_type': 'low', # or 'high'
|
|
'confidence': 0.85
|
|
}
|
|
}
|
|
```
|
|
|
|
## Monitoring
|
|
|
|
### Log Messages
|
|
|
|
**Startup:**
|
|
```
|
|
LivePivotTrainer initialized
|
|
LivePivotTrainer started for ETH/USDT
|
|
✅ Live pivot training ENABLED - will train on L2 peaks automatically
|
|
```
|
|
|
|
**Pivot Detection:**
|
|
```
|
|
Found 2 new L2 pivots on ETH/USDT 1m
|
|
Training on L2 low pivot @ 3150.50 on ETH/USDT 1m
|
|
Started background training on L2 pivot
|
|
Live pivot training session started: abc-123-def
|
|
```
|
|
|
|
**Statistics:**
|
|
```python
|
|
stats = pivot_trainer.get_stats()
|
|
# {
|
|
# 'running': True,
|
|
# 'total_trained_pivots': 47,
|
|
# 'last_training_1s': 1699876543.21,
|
|
# 'last_training_1m': 1699876540.15,
|
|
# 'pivot_history_1s': 100,
|
|
# 'pivot_history_1m': 100
|
|
# }
|
|
```
|
|
|
|
## Performance Considerations
|
|
|
|
### Memory Usage
|
|
- Tracks last 1000 trained pivots (~50KB)
|
|
- Pivot history: 100 per timeframe (~10KB)
|
|
- **Total overhead: <100KB**
|
|
|
|
### CPU Usage
|
|
- Checks every 5 seconds (configurable)
|
|
- Pivot detection: ~10ms per check
|
|
- **Minimal impact on inference**
|
|
|
|
### Training Frequency
|
|
- Rate limited: 60 seconds between training on same timeframe
|
|
- Prevents overtraining on noisy pivots
|
|
- Typical: 2-10 training sessions per hour
|
|
|
|
## Best Practices
|
|
|
|
### 1. Start with Default Settings
|
|
```python
|
|
# Let it run with defaults first
|
|
pivot_trainer.start(symbol='ETH/USDT')
|
|
```
|
|
|
|
### 2. Monitor Training Quality
|
|
```python
|
|
# Check how many pivots are being trained on
|
|
stats = pivot_trainer.get_stats()
|
|
if stats['total_trained_pivots'] > 100:
|
|
# Increase min_pivot_spacing to reduce frequency
|
|
pivot_trainer.min_pivot_spacing = 120 # 2 minutes
|
|
```
|
|
|
|
### 3. Adjust for Market Conditions
|
|
```python
|
|
# Volatile market - train more frequently
|
|
pivot_trainer.min_pivot_spacing = 30 # 30 seconds
|
|
|
|
# Quiet market - train less frequently
|
|
pivot_trainer.min_pivot_spacing = 300 # 5 minutes
|
|
```
|
|
|
|
### 4. Combine with Manual Annotations
|
|
- Live pivot training handles routine patterns
|
|
- Manual annotations for special cases
|
|
- Best of both worlds!
|
|
|
|
## Troubleshooting
|
|
|
|
### No Pivots Detected
|
|
**Problem:** `total_trained_pivots` stays at 0
|
|
|
|
**Solutions:**
|
|
1. Check if Williams Market Structure is initialized:
|
|
```python
|
|
if pivot_trainer.williams_1s is None:
|
|
# Reinstall/fix Williams Market Structure
|
|
```
|
|
|
|
2. Verify data is flowing:
|
|
```python
|
|
candles = data_provider.get_historical_data('ETH/USDT', '1m', 200)
|
|
print(f"Candles: {len(candles)}")
|
|
```
|
|
|
|
3. Lower pivot detection threshold (if available)
|
|
|
|
### Too Many Training Sessions
|
|
**Problem:** Training every few seconds, slowing down system
|
|
|
|
**Solution:**
|
|
```python
|
|
# Increase spacing
|
|
pivot_trainer.min_pivot_spacing = 180 # 3 minutes
|
|
|
|
# Or reduce check frequency
|
|
pivot_trainer.check_interval = 10 # Check every 10 seconds
|
|
```
|
|
|
|
### Training Errors
|
|
**Problem:** Background training fails
|
|
|
|
**Check logs:**
|
|
```
|
|
Error in background training: ...
|
|
```
|
|
|
|
**Solutions:**
|
|
1. Verify training adapter is working:
|
|
```python
|
|
# Test manual training first
|
|
training_adapter.start_training('Transformer', [test_case])
|
|
```
|
|
|
|
2. Check memory availability (training needs RAM)
|
|
|
|
3. Verify model is loaded in orchestrator
|
|
|
|
## Integration with Existing Systems
|
|
|
|
### Works With:
|
|
- ✅ Manual annotation training
|
|
- ✅ Real-time inference
|
|
- ✅ All model types (Transformer, CNN, DQN)
|
|
- ✅ Multiple symbols (start separate instances)
|
|
|
|
### Doesn't Interfere With:
|
|
- ✅ Live inference predictions
|
|
- ✅ Manual training sessions
|
|
- ✅ Checkpoint saving/loading
|
|
- ✅ Dashboard updates
|
|
|
|
## Future Enhancements
|
|
|
|
### Planned Features:
|
|
1. **Adaptive Spacing** - Adjust `min_pivot_spacing` based on market volatility
|
|
2. **Multi-Level Training** - Train on L1, L2, L3 pivots with different priorities
|
|
3. **Outcome Tracking** - Track actual profit/loss of pivot-based trades
|
|
4. **Quality Filtering** - Only train on high-confidence pivots
|
|
5. **Multi-Symbol** - Monitor multiple symbols simultaneously
|
|
|
|
### Configuration UI (Future):
|
|
```
|
|
┌─────────────────────────────────────┐
|
|
│ Live Pivot Training Settings │
|
|
├─────────────────────────────────────┤
|
|
│ ☑ Enable Auto-Training │
|
|
│ │
|
|
│ Timeframes: │
|
|
│ ☑ 1s ☑ 1m ☐ 1h ☐ 1d │
|
|
│ │
|
|
│ Min Spacing: [60] seconds │
|
|
│ Check Interval: [5] seconds │
|
|
│ │
|
|
│ Pivot Levels: │
|
|
│ ☐ L1 ☑ L2 ☐ L3 │
|
|
│ │
|
|
│ Stats: │
|
|
│ Trained: 47 pivots │
|
|
│ Last 1s: 2 min ago │
|
|
│ Last 1m: 5 min ago │
|
|
└─────────────────────────────────────┘
|
|
```
|
|
|
|
## Summary
|
|
|
|
Live Pivot Training provides **automatic, continuous learning** from live market data by:
|
|
1. Detecting significant L2 pivot points
|
|
2. Creating training samples automatically
|
|
3. Training models in background
|
|
4. Adapting to current market conditions
|
|
|
|
**Result:** Your models continuously improve without manual intervention!
|