wip training

.kiro/specs/multi-modal-trading-system/design.md

@@ -140,7 +140,7 @@ Training:
 
 ### 4. Orchestrator
 
-The Orchestrator serves as the central coordination hub of the multi-modal trading system, responsible for data subscription management, model inference coordination, output storage, and training pipeline orchestration.
+The Orchestrator serves as the central coordination hub of the multi-modal trading system, responsible for data subscription management, model inference coordination, output storage, training pipeline orchestration, and inference-training feedback loop management.
 
 #### Key Classes and Interfaces
 
@@ -245,6 +245,47 @@ The Orchestrator coordinates training for all models by managing the prediction-
 - checkpoint manager has capability to ensure only top 5 to 10 best checkpoints are stored for each model deleting the least performant ones. it stores metadata along the CPs to decide the performance
 - we automatically load the best CP at startup if we have stored ones
 
+##### 5. Inference Data Validation and Storage
+
+The Orchestrator implements comprehensive inference data validation and persistent storage:
+
+**Input Data Validation**:
+- Validates complete OHLCV dataframes for all required timeframes before inference
+- Checks input data dimensions against model requirements
+- Logs missing components and prevents prediction on incomplete data
+- Raises validation errors with specific details about expected vs actual dimensions
+
+**Inference History Storage**:
+- Stores complete input data packages with each prediction in persistent storage
+- Includes timestamp, symbol, input features, prediction outputs, confidence scores, and model internal states
+- Maintains compressed storage to minimize footprint while preserving accessibility
+- Implements efficient query mechanisms by symbol, timeframe, and date range
+
+**Storage Management**:
+- Applies configurable retention policies to manage storage limits
+- Archives or removes oldest entries when limits are reached
+- Prioritizes keeping most recent and valuable training examples during storage pressure
+- Provides data completeness metrics and validation results in logs
+
+##### 6. Inference-Training Feedback Loop
+
+The Orchestrator manages the continuous learning cycle through inference-training feedback:
+
+**Prediction Outcome Evaluation**:
+- Evaluates prediction accuracy against actual price movements after sufficient time has passed
+- Creates training examples using stored inference data paired with actual market outcomes
+- Feeds prediction-result pairs back to respective models for learning
+
+**Adaptive Learning Signals**:
+- Provides positive reinforcement signals for accurate predictions
+- Delivers corrective training signals for inaccurate predictions to help models learn from mistakes
+- Retrieves last inference data for each model to compare predictions against actual outcomes
+
+**Continuous Improvement Tracking**:
+- Tracks and reports accuracy improvements or degradations over time
+- Monitors model learning progress through the feedback loop
+- Alerts administrators when data flow issues are detected with specific error details and remediation suggestions
+
 ##### 5. Decision Making and Trading Actions
 
 Beyond coordination, the Orchestrator makes final trading decisions: