fetching data from the DB to train
This commit is contained in:
Dobromir Popov
333
ANNOTATE/CONTINUOUS_DATA_TRAINING_STRATEGY.md
Normal file
333
ANNOTATE/CONTINUOUS_DATA_TRAINING_STRATEGY.md
Normal file
@@ -0,0 +1,333 @@
|
||||
# Continuous Data Training Strategy
|
||||
|
||||
## Overview
|
||||
|
||||
The ANNOTATE system trains models on **continuous OHLCV data** from the database, not just on annotated signals. This teaches the model **when to act AND when NOT to act**.
|
||||
|
||||
## Training Data Composition
|
||||
|
||||
For each annotation, the system creates multiple training samples:
|
||||
|
||||
### 1. ENTRY Sample (1 per annotation)
|
||||
- **Label**: `ENTRY`
|
||||
- **Action**: `BUY` or `SELL`
|
||||
- **Purpose**: Teach model to recognize entry signals
|
||||
- **Repetitions**: 100x (configurable)
|
||||
|
||||
```python
|
||||
{
|
||||
'label': 'ENTRY',
|
||||
'action': 'BUY',
|
||||
'direction': 'LONG',
|
||||
'timestamp': '2025-10-27 14:00',
|
||||
'entry_price': 2500.0,
|
||||
'repetitions': 100
|
||||
}
|
||||
```
|
||||
|
||||
### 2. HOLD Samples (N per annotation)
|
||||
- **Label**: `HOLD`
|
||||
- **Action**: `HOLD`
|
||||
- **Purpose**: Teach model to maintain position
|
||||
- **Count**: Every candle between entry and exit
|
||||
- **Repetitions**: 25x (1/4 of entry reps)
|
||||
|
||||
```python
|
||||
# For a 30-minute trade with 1m candles = 30 HOLD samples
|
||||
{
|
||||
'label': 'HOLD',
|
||||
'action': 'HOLD',
|
||||
'in_position': True,
|
||||
'timestamp': '2025-10-27 14:05', # During position
|
||||
'repetitions': 25
|
||||
}
|
||||
```
|
||||
|
||||
### 3. EXIT Sample (1 per annotation)
|
||||
- **Label**: `EXIT`
|
||||
- **Action**: `CLOSE`
|
||||
- **Purpose**: Teach model to recognize exit signals
|
||||
- **Repetitions**: 100x
|
||||
|
||||
```python
|
||||
{
|
||||
'label': 'EXIT',
|
||||
'action': 'CLOSE',
|
||||
'timestamp': '2025-10-27 14:30',
|
||||
'exit_price': 2562.5,
|
||||
'profit_loss_pct': 2.5,
|
||||
'repetitions': 100
|
||||
}
|
||||
```
|
||||
|
||||
### 4. NO_TRADE Samples (±15 candles per annotation)
|
||||
- **Label**: `NO_TRADE`
|
||||
- **Action**: `HOLD`
|
||||
- **Purpose**: Teach model when NOT to trade
|
||||
- **Count**: Up to 30 samples (15 before + 15 after signal)
|
||||
- **Repetitions**: 50x (1/2 of entry reps)
|
||||
|
||||
```python
|
||||
# 15 candles BEFORE entry signal
|
||||
{
|
||||
'label': 'NO_TRADE',
|
||||
'action': 'HOLD',
|
||||
'timestamp': '2025-10-27 13:45', # 15 min before entry
|
||||
'direction': 'NONE',
|
||||
'repetitions': 50
|
||||
}
|
||||
|
||||
# 15 candles AFTER entry signal
|
||||
{
|
||||
'label': 'NO_TRADE',
|
||||
'action': 'HOLD',
|
||||
'timestamp': '2025-10-27 14:15', # 15 min after entry
|
||||
'direction': 'NONE',
|
||||
'repetitions': 50
|
||||
}
|
||||
```
|
||||
|
||||
## Data Fetching Strategy
|
||||
|
||||
### Extended Time Window
|
||||
|
||||
To support negative sampling (±15 candles), the system fetches an **extended time window**:
|
||||
|
||||
```python
|
||||
# Configuration
|
||||
context_window_minutes = 5 # Base context
|
||||
negative_samples_window = 15 # ±15 candles
|
||||
extended_window = max(5, 15 + 10) # = 25 minutes
|
||||
|
||||
# Time range
|
||||
start_time = entry_timestamp - 25 minutes
|
||||
end_time = entry_timestamp + 25 minutes
|
||||
```
|
||||
|
||||
### Candle Limits by Timeframe
|
||||
|
||||
```python
|
||||
# 1s timeframe: 25 min × 60 sec × 2 + buffer = ~3100 candles
|
||||
# 1m timeframe: 25 min × 2 + buffer = ~100 candles
|
||||
# 1h timeframe: 200 candles (fixed)
|
||||
# 1d timeframe: 200 candles (fixed)
|
||||
```
|
||||
|
||||
## Training Sample Distribution
|
||||
|
||||
### Example: Single Annotation
|
||||
|
||||
```
|
||||
Annotation: LONG entry at 14:00, exit at 14:30 (30 min hold)
|
||||
|
||||
Training Samples Created:
|
||||
├── 1 ENTRY sample @ 14:00 (×100 reps) = 100 batches
|
||||
├── 30 HOLD samples @ 14:01-14:29 (×25 reps) = 750 batches
|
||||
├── 1 EXIT sample @ 14:30 (×100 reps) = 100 batches
|
||||
└── 30 NO_TRADE samples @ 13:45-13:59 & 14:01-14:15 (×50 reps) = 1500 batches
|
||||
|
||||
Total: 62 unique samples → 2,450 training batches
|
||||
```
|
||||
|
||||
### Example: 5 Annotations
|
||||
|
||||
```
|
||||
5 annotations with similar structure:
|
||||
|
||||
Training Samples:
|
||||
├── ENTRY: 5 samples (×100 reps) = 500 batches
|
||||
├── HOLD: ~150 samples (×25 reps) = 3,750 batches
|
||||
├── EXIT: 5 samples (×100 reps) = 500 batches
|
||||
└── NO_TRADE: ~150 samples (×50 reps) = 7,500 batches
|
||||
|
||||
Total: ~310 unique samples → 12,250 training batches
|
||||
|
||||
Ratio: 1:30 (entry:no_trade) - teaches model to be selective!
|
||||
```
|
||||
|
||||
## Why This Works
|
||||
|
||||
### 1. Reduces False Positives
|
||||
By training on NO_TRADE samples around signals, the model learns:
|
||||
- Not every price movement is a signal
|
||||
- Context matters (what happened before/after)
|
||||
- Patience is important (wait for the right moment)
|
||||
|
||||
### 2. Improves Timing
|
||||
By training on continuous data, the model learns:
|
||||
- Gradual buildup to entry signals
|
||||
- How market conditions evolve
|
||||
- Difference between "almost" and "ready"
|
||||
|
||||
### 3. Teaches Position Management
|
||||
By training on HOLD samples, the model learns:
|
||||
- When to stay in position
|
||||
- Not to exit early
|
||||
- How to ride trends
|
||||
|
||||
### 4. Balanced Training
|
||||
The repetition strategy ensures balanced learning:
|
||||
- ENTRY: 100 reps (high importance)
|
||||
- EXIT: 100 reps (high importance)
|
||||
- NO_TRADE: 50 reps (moderate importance)
|
||||
- HOLD: 25 reps (lower importance, but many samples)
|
||||
|
||||
## Database Requirements
|
||||
|
||||
### Continuous OHLCV Storage
|
||||
|
||||
The system requires **continuous historical data** in DuckDB:
|
||||
|
||||
```sql
|
||||
-- Example: Check data availability
|
||||
SELECT
|
||||
symbol,
|
||||
timeframe,
|
||||
COUNT(*) as candle_count,
|
||||
MIN(timestamp) as first_candle,
|
||||
MAX(timestamp) as last_candle
|
||||
FROM ohlcv_data
|
||||
WHERE symbol = 'ETH/USDT'
|
||||
GROUP BY symbol, timeframe;
|
||||
```
|
||||
|
||||
### Data Gaps
|
||||
|
||||
If there are gaps in the data:
|
||||
- Negative samples will be fewer (< 30)
|
||||
- Model still trains but with less context
|
||||
- Warning logged: "Could not create full negative sample set"
|
||||
|
||||
## Configuration
|
||||
|
||||
### Adjustable Parameters
|
||||
|
||||
```python
|
||||
# In _prepare_training_data()
|
||||
negative_samples_window = 15 # ±15 candles (default)
|
||||
training_repetitions = 100 # 100x per sample (default)
|
||||
|
||||
# Derived repetitions
|
||||
hold_repetitions = 100 // 4 # 25x
|
||||
no_trade_repetitions = 100 // 2 # 50x
|
||||
```
|
||||
|
||||
### Tuning Guidelines
|
||||
|
||||
| Parameter | Small Dataset | Large Dataset | High Precision |
|
||||
|-----------|--------------|---------------|----------------|
|
||||
| `negative_samples_window` | 10 | 20 | 15 |
|
||||
| `training_repetitions` | 50 | 200 | 100 |
|
||||
| `extended_window_minutes` | 15 | 30 | 25 |
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Training Logs
|
||||
|
||||
Look for these log messages:
|
||||
|
||||
```
|
||||
✅ Good:
|
||||
"Fetching HISTORICAL market state for ETH/USDT at 2025-10-27 14:00"
|
||||
"Extended window: ±25 minutes (Includes ±15 candles for negative sampling)"
|
||||
"1m: 100 candles from DuckDB (historical)"
|
||||
"Added 30 NO_TRADE samples (±15 candles)"
|
||||
"→ 15 before signal, 15 after signal"
|
||||
|
||||
⚠️ Warning:
|
||||
"No historical data found, using latest data as fallback"
|
||||
"Could not create full negative sample set (only 8 samples)"
|
||||
"Market data has 50 timestamps from ... to ..." (insufficient data)
|
||||
```
|
||||
|
||||
### Sample Distribution
|
||||
|
||||
Check the final distribution:
|
||||
|
||||
```
|
||||
INFO - Prepared 310 training samples from 5 test cases
|
||||
INFO - ENTRY samples: 5
|
||||
INFO - HOLD samples: 150
|
||||
INFO - EXIT samples: 5
|
||||
INFO - NO_TRADE samples: 150
|
||||
INFO - Ratio: 1:30.0 (entry:no_trade)
|
||||
```
|
||||
|
||||
**Ideal Ratio**: 1:20 to 1:40 (entry:no_trade)
|
||||
- Too low (< 1:10): Model may overtrade
|
||||
- Too high (> 1:50): Model may undertrade
|
||||
|
||||
## Benefits
|
||||
|
||||
### 1. Realistic Training
|
||||
- Trains on actual market conditions
|
||||
- Includes noise and false signals
|
||||
- Learns from continuous price action
|
||||
|
||||
### 2. Better Generalization
|
||||
- Not just memorizing entry points
|
||||
- Understands context and timing
|
||||
- Reduces overfitting
|
||||
|
||||
### 3. Selective Trading
|
||||
- High ratio of NO_TRADE samples
|
||||
- Learns to wait for quality setups
|
||||
- Reduces false signals in production
|
||||
|
||||
### 4. Efficient Use of Data
|
||||
- One annotation → 60+ training samples
|
||||
- Leverages continuous database storage
|
||||
- No manual labeling of negative samples
|
||||
|
||||
## Example Training Session
|
||||
|
||||
```
|
||||
Starting REAL training with 5 test cases for model Transformer
|
||||
|
||||
Preparing training data from 5 test cases...
|
||||
Negative sampling: +/-15 candles around signals
|
||||
Training repetitions: 100x per sample
|
||||
|
||||
Fetching market state dynamically for test case 1...
|
||||
Fetching HISTORICAL market state for ETH/USDT at 2025-10-27 14:00
|
||||
Timeframes: ['1s', '1m', '1h', '1d'], Extended window: ±25 minutes
|
||||
(Includes ±15 candles for negative sampling)
|
||||
1m: 100 candles from DuckDB (historical)
|
||||
1h: 200 candles from DuckDB (historical)
|
||||
1d: 200 candles from DuckDB (historical)
|
||||
Fetched market state with 3 timeframes
|
||||
|
||||
Test case 1: ENTRY sample - LONG @ 2500.0
|
||||
Test case 1: Added 30 HOLD samples (during position)
|
||||
Test case 1: EXIT sample @ 2562.5 (2.50%)
|
||||
Test case 1: Added 30 NO_TRADE samples (±15 candles)
|
||||
→ 15 before signal, 15 after signal
|
||||
|
||||
[... repeat for test cases 2-5 ...]
|
||||
|
||||
Prepared 310 training samples from 5 test cases
|
||||
ENTRY samples: 5
|
||||
HOLD samples: 150
|
||||
EXIT samples: 5
|
||||
NO_TRADE samples: 150
|
||||
Ratio: 1:30.0 (entry:no_trade)
|
||||
|
||||
Starting Transformer training...
|
||||
Converting annotation data to transformer format...
|
||||
Converted 310 samples to 12,250 training batches
|
||||
|
||||
Training batch 1/12250: loss=0.523
|
||||
Training batch 100/12250: loss=0.412
|
||||
Training batch 200/12250: loss=0.356
|
||||
...
|
||||
```
|
||||
|
||||
## Summary
|
||||
|
||||
- ✅ Trains on **continuous OHLCV data** from database
|
||||
- ✅ Creates **±15 candle negative samples** automatically
|
||||
- ✅ Teaches model **when to act AND when NOT to act**
|
||||
- ✅ Uses **extended time window** to fetch sufficient data
|
||||
- ✅ Balanced training with **1:30 entry:no_trade ratio**
|
||||
- ✅ Efficient: **1 annotation → 60+ training samples**
|
||||
- ✅ Realistic: Includes noise, false signals, and context
|
||||
Reference in New Issue
Block a user