Files

Dobromir Popov 25287d0e9e kiro steering, live training wip

2025-11-13 15:09:20 +02:00

8.7 KiB

Raw Permalink Blame History

Live Pivot Training - Automatic Training on Market Structure

Overview

The Live Pivot Training system automatically trains your models on significant market structure points (L2 pivots) detected in real-time on 1s and 1m charts.

Key Benefits:

✅ Continuous learning from live market data
✅ Trains on high-quality pivot points (peaks and troughs)
✅ Non-blocking - doesn't interfere with inference
✅ Automatic - no manual annotation needed
✅ Adaptive - learns from current market conditions

How It Works

1. Pivot Detection

Live Market Data (1s, 1m)
    ↓
Williams Market Structure
    ↓
L2 Pivot Detection
    ↓
High/Low Identification

2. Training Sample Creation

When an L2 pivot is detected:

High Pivot → Creates SHORT training sample
Low Pivot → Creates LONG training sample

3. Background Training

Training happens in separate thread
Doesn't block inference
Uses same training pipeline as manual annotations

Usage

Starting Live Inference with Auto-Training

Default (Auto-training ENABLED):

fetch('/api/realtime-inference/start', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
        model_name: 'Transformer',
        symbol: 'ETH/USDT'
        // enable_live_training: true (default)
    })
})

Disable Auto-Training:

fetch('/api/realtime-inference/start', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
        model_name: 'Transformer',
        symbol: 'ETH/USDT',
        enable_live_training: false  // Disable
    })
})

Python API

from ANNOTATE.core.live_pivot_trainer import get_live_pivot_trainer

# Get trainer instance
pivot_trainer = get_live_pivot_trainer(
    orchestrator=orchestrator,
    data_provider=data_provider,
    training_adapter=training_adapter
)

# Start monitoring
pivot_trainer.start(symbol='ETH/USDT')

# Get statistics
stats = pivot_trainer.get_stats()
print(f"Trained on {stats['total_trained_pivots']} pivots")

# Stop monitoring
pivot_trainer.stop()

Configuration

Adjustable Parameters

Located in ANNOTATE/core/live_pivot_trainer.py:

class LivePivotTrainer:
    def __init__(self, ...):
        # Check for new pivots every 5 seconds
        self.check_interval = 5
        
        # Minimum 60 seconds between training on same timeframe
        self.min_pivot_spacing = 60
        
        # Track last 1000 trained pivots (avoid duplicates)
        self.trained_pivots = deque(maxlen=1000)

Timeframe Configuration

Currently monitors:

1s timeframe - High-frequency pivots
1m timeframe - Short-term pivots

Can be extended to 1h, 1d by modifying _monitoring_loop().

Training Sample Structure

Each L2 pivot generates a training sample:

{
    'test_case_id': 'live_pivot_ETH/USDT_1m_2025-11-12T18:30:00',
    'symbol': 'ETH/USDT',
    'timestamp': '2025-11-12T18:30:00+00:00',
    'action': 'BUY',  # or 'SELL' for high pivots
    'expected_outcome': {
        'direction': 'LONG',  # or 'SHORT'
        'entry_price': 3150.50,
        'exit_price': None,  # Determined by model
        'profit_loss_pct': 0.0,
        'holding_period_seconds': 300  # 5 minutes default
    },
    'training_config': {
        'timeframes': ['1s', '1m', '1h', '1d'],
        'candles_per_timeframe': 200
    },
    'annotation_metadata': {
        'source': 'live_pivot_detection',
        'pivot_level': 'L2',
        'pivot_type': 'low',  # or 'high'
        'confidence': 0.85
    }
}

Monitoring

Log Messages

Startup:

LivePivotTrainer initialized
LivePivotTrainer started for ETH/USDT
✅ Live pivot training ENABLED - will train on L2 peaks automatically

Pivot Detection:

Found 2 new L2 pivots on ETH/USDT 1m
Training on L2 low pivot @ 3150.50 on ETH/USDT 1m
Started background training on L2 pivot
Live pivot training session started: abc-123-def

Statistics:

stats = pivot_trainer.get_stats()
# {
#     'running': True,
#     'total_trained_pivots': 47,
#     'last_training_1s': 1699876543.21,
#     'last_training_1m': 1699876540.15,
#     'pivot_history_1s': 100,
#     'pivot_history_1m': 100
# }

Performance Considerations

Memory Usage

Tracks last 1000 trained pivots (~50KB)
Pivot history: 100 per timeframe (~10KB)
Total overhead: <100KB

CPU Usage

Checks every 5 seconds (configurable)
Pivot detection: ~10ms per check
Minimal impact on inference

Training Frequency

Rate limited: 60 seconds between training on same timeframe
Prevents overtraining on noisy pivots
Typical: 2-10 training sessions per hour

Best Practices

1. Start with Default Settings

# Let it run with defaults first
pivot_trainer.start(symbol='ETH/USDT')

2. Monitor Training Quality

# Check how many pivots are being trained on
stats = pivot_trainer.get_stats()
if stats['total_trained_pivots'] > 100:
    # Increase min_pivot_spacing to reduce frequency
    pivot_trainer.min_pivot_spacing = 120  # 2 minutes

3. Adjust for Market Conditions

# Volatile market - train more frequently
pivot_trainer.min_pivot_spacing = 30  # 30 seconds

# Quiet market - train less frequently
pivot_trainer.min_pivot_spacing = 300  # 5 minutes

4. Combine with Manual Annotations

Live pivot training handles routine patterns
Manual annotations for special cases
Best of both worlds!

Troubleshooting

No Pivots Detected

Problem: total_trained_pivots stays at 0

Solutions:

Check if Williams Market Structure is initialized:

if pivot_trainer.williams_1s is None:
    # Reinstall/fix Williams Market Structure

Verify data is flowing:

candles = data_provider.get_historical_data('ETH/USDT', '1m', 200)
print(f"Candles: {len(candles)}")

Lower pivot detection threshold (if available)

Too Many Training Sessions

Problem: Training every few seconds, slowing down system

Solution:

# Increase spacing
pivot_trainer.min_pivot_spacing = 180  # 3 minutes

# Or reduce check frequency
pivot_trainer.check_interval = 10  # Check every 10 seconds

Training Errors

Problem: Background training fails

Check logs:

Error in background training: ...