Files

Dobromir Popov 6ac324289c fetching data from the DB to train

2025-10-31 03:14:35 +02:00

8.8 KiB

Raw Blame History

Continuous Data Training Strategy

Overview

The ANNOTATE system trains models on continuous OHLCV data from the database, not just on annotated signals. This teaches the model when to act AND when NOT to act.

Training Data Composition

For each annotation, the system creates multiple training samples:

1. ENTRY Sample (1 per annotation)

Label: ENTRY
Action: BUY or SELL
Purpose: Teach model to recognize entry signals
Repetitions: 100x (configurable)

{
    'label': 'ENTRY',
    'action': 'BUY',
    'direction': 'LONG',
    'timestamp': '2025-10-27 14:00',
    'entry_price': 2500.0,
    'repetitions': 100
}

2. HOLD Samples (N per annotation)

Label: HOLD
Action: HOLD
Purpose: Teach model to maintain position
Count: Every candle between entry and exit
Repetitions: 25x (1/4 of entry reps)

# For a 30-minute trade with 1m candles = 30 HOLD samples
{
    'label': 'HOLD',
    'action': 'HOLD',
    'in_position': True,
    'timestamp': '2025-10-27 14:05',  # During position
    'repetitions': 25
}

3. EXIT Sample (1 per annotation)

Label: EXIT
Action: CLOSE
Purpose: Teach model to recognize exit signals
Repetitions: 100x

{
    'label': 'EXIT',
    'action': 'CLOSE',
    'timestamp': '2025-10-27 14:30',
    'exit_price': 2562.5,
    'profit_loss_pct': 2.5,
    'repetitions': 100
}

4. NO_TRADE Samples (±15 candles per annotation)

Label: NO_TRADE
Action: HOLD
Purpose: Teach model when NOT to trade
Count: Up to 30 samples (15 before + 15 after signal)
Repetitions: 50x (1/2 of entry reps)

# 15 candles BEFORE entry signal
{
    'label': 'NO_TRADE',
    'action': 'HOLD',
    'timestamp': '2025-10-27 13:45',  # 15 min before entry
    'direction': 'NONE',
    'repetitions': 50
}

# 15 candles AFTER entry signal
{
    'label': 'NO_TRADE',
    'action': 'HOLD',
    'timestamp': '2025-10-27 14:15',  # 15 min after entry
    'direction': 'NONE',
    'repetitions': 50
}

Data Fetching Strategy

Extended Time Window

To support negative sampling (±15 candles), the system fetches an extended time window:

# Configuration
context_window_minutes = 5          # Base context
negative_samples_window = 15        # ±15 candles
extended_window = max(5, 15 + 10)   # = 25 minutes

# Time range
start_time = entry_timestamp - 25 minutes
end_time = entry_timestamp + 25 minutes

Candle Limits by Timeframe

# 1s timeframe: 25 min × 60 sec × 2 + buffer = ~3100 candles
# 1m timeframe: 25 min × 2 + buffer = ~100 candles
# 1h timeframe: 200 candles (fixed)
# 1d timeframe: 200 candles (fixed)

Training Sample Distribution

Example: Single Annotation

Annotation: LONG entry at 14:00, exit at 14:30 (30 min hold)

Training Samples Created:
├── 1 ENTRY sample @ 14:00 (×100 reps) = 100 batches
├── 30 HOLD samples @ 14:01-14:29 (×25 reps) = 750 batches
├── 1 EXIT sample @ 14:30 (×100 reps) = 100 batches
└── 30 NO_TRADE samples @ 13:45-13:59 & 14:01-14:15 (×50 reps) = 1500 batches

Total: 62 unique samples → 2,450 training batches

Example: 5 Annotations

5 annotations with similar structure:

Training Samples:
├── ENTRY: 5 samples (×100 reps) = 500 batches
├── HOLD: ~150 samples (×25 reps) = 3,750 batches
├── EXIT: 5 samples (×100 reps) = 500 batches
└── NO_TRADE: ~150 samples (×50 reps) = 7,500 batches

Total: ~310 unique samples → 12,250 training batches

Ratio: 1:30 (entry:no_trade) - teaches model to be selective!

Why This Works

1. Reduces False Positives

By training on NO_TRADE samples around signals, the model learns:

Not every price movement is a signal
Context matters (what happened before/after)
Patience is important (wait for the right moment)

2. Improves Timing

By training on continuous data, the model learns:

Gradual buildup to entry signals
How market conditions evolve
Difference between "almost" and "ready"

3. Teaches Position Management

By training on HOLD samples, the model learns:

When to stay in position
Not to exit early
How to ride trends

4. Balanced Training

The repetition strategy ensures balanced learning:

ENTRY: 100 reps (high importance)
EXIT: 100 reps (high importance)
NO_TRADE: 50 reps (moderate importance)
HOLD: 25 reps (lower importance, but many samples)

Database Requirements

Continuous OHLCV Storage

The system requires continuous historical data in DuckDB:

-- Example: Check data availability
SELECT 
    symbol,
    timeframe,
    COUNT(*) as candle_count,
    MIN(timestamp) as first_candle,
    MAX(timestamp) as last_candle
FROM ohlcv_data
WHERE symbol = 'ETH/USDT'
GROUP BY symbol, timeframe;

Data Gaps

If there are gaps in the data:

Negative samples will be fewer (< 30)
Model still trains but with less context
Warning logged: "Could not create full negative sample set"

Configuration

Adjustable Parameters

# In _prepare_training_data()
negative_samples_window = 15      # ±15 candles (default)
training_repetitions = 100        # 100x per sample (default)

# Derived repetitions
hold_repetitions = 100 // 4       # 25x
no_trade_repetitions = 100 // 2   # 50x

Tuning Guidelines

Parameter	Small Dataset	Large Dataset	High Precision
`negative_samples_window`	10	20	15
`training_repetitions`	50	200	100
`extended_window_minutes`	15	30	25

Monitoring

Training Logs

Look for these log messages:

✅ Good:
"Fetching HISTORICAL market state for ETH/USDT at 2025-10-27 14:00"
"Extended window: ±25 minutes (Includes ±15 candles for negative sampling)"
"1m: 100 candles from DuckDB (historical)"
"Added 30 NO_TRADE samples (±15 candles)"
"→ 15 before signal, 15 after signal"

⚠️ Warning:
"No historical data found, using latest data as fallback"
"Could not create full negative sample set (only 8 samples)"
"Market data has 50 timestamps from ... to ..." (insufficient data)

Sample Distribution

Check the final distribution:

INFO - Prepared 310 training samples from 5 test cases
INFO -    ENTRY samples: 5
INFO -    HOLD samples: 150
INFO -    EXIT samples: 5
INFO -    NO_TRADE samples: 150
INFO -    Ratio: 1:30.0 (entry:no_trade)

Ideal Ratio: 1:20 to 1:40 (entry:no_trade)

Too low (< 1:10): Model may overtrade
Too high (> 1:50): Model may undertrade

Benefits

1. Realistic Training

Trains on actual market conditions
Includes noise and false signals
Learns from continuous price action

2. Better Generalization

Not just memorizing entry points
Understands context and timing
Reduces overfitting

3. Selective Trading

High ratio of NO_TRADE samples
Learns to wait for quality setups
Reduces false signals in production

4. Efficient Use of Data

One annotation → 60+ training samples
Leverages continuous database storage
No manual labeling of negative samples

Example Training Session

Starting REAL training with 5 test cases for model Transformer

Preparing training data from 5 test cases...
   Negative sampling: +/-15 candles around signals
   Training repetitions: 100x per sample

   Fetching market state dynamically for test case 1...
   Fetching HISTORICAL market state for ETH/USDT at 2025-10-27 14:00
   Timeframes: ['1s', '1m', '1h', '1d'], Extended window: ±25 minutes
   (Includes ±15 candles for negative sampling)
       1m: 100 candles from DuckDB (historical)
       1h: 200 candles from DuckDB (historical)
       1d: 200 candles from DuckDB (historical)
    Fetched market state with 3 timeframes

   Test case 1: ENTRY sample - LONG @ 2500.0
   Test case 1: Added 30 HOLD samples (during position)
   Test case 1: EXIT sample @ 2562.5 (2.50%)
   Test case 1: Added 30 NO_TRADE samples (±15 candles)
      → 15 before signal, 15 after signal

[... repeat for test cases 2-5 ...]

 Prepared 310 training samples from 5 test cases
   ENTRY samples: 5
   HOLD samples: 150
   EXIT samples: 5
   NO_TRADE samples: 150
   Ratio: 1:30.0 (entry:no_trade)

 Starting Transformer training...
    Converting annotation data to transformer format...
     Converted 310 samples to 12,250 training batches

Training batch 1/12250: loss=0.523
Training batch 100/12250: loss=0.412
Training batch 200/12250: loss=0.356
...

Summary

✅ Trains on continuous OHLCV data from database
✅ Creates ±15 candle negative samples automatically
✅ Teaches model when to act AND when NOT to act
✅ Uses extended time window to fetch sufficient data
✅ Balanced training with 1:30 entry:no_trade ratio
✅ Efficient: 1 annotation → 60+ training samples
✅ Realistic: Includes noise, false signals, and context

8.8 KiB Raw Blame History Unescape Escape