gogo2/.kiro/specs/checkpoint-persistence-fix/requirements.md

Requirements Document

Introduction

The Checkpoint Persistence Fix addresses a critical system flaw where model training progress is not being saved during training, causing all learning progress to be lost when the system restarts. Despite having a well-implemented CheckpointManager and proper checkpoint loading at startup, the system lacks checkpoint saving during training operations. This creates a fundamental issue where models train continuously but never persist their improved weights, making continuous improvement impossible and wasting computational resources.

Requirements

Requirement 1: Real-time Checkpoint Saving During Training

User Story: As a system operator, I want model improvements to be automatically saved during training, so that training progress is never lost when the system restarts.

Acceptance Criteria

1. WHEN the DQN model is trained in _train_models_on_decision THEN the system SHALL save a checkpoint if the loss improves.
2. WHEN the CNN model is trained THEN the system SHALL save a checkpoint if the loss improves.
3. WHEN the COB RL model is trained THEN the system SHALL save a checkpoint if the loss improves.
4. WHEN the Extrema trainer is trained THEN the system SHALL save a checkpoint if the loss improves.
5. WHEN any model training completes THEN the system SHALL compare current performance to best performance and save if improved.
6. WHEN checkpoint saving occurs THEN the system SHALL update the model_states dictionary with new performance metrics.

Requirement 2: Performance-Based Checkpoint Management

User Story: As a developer, I want checkpoints to be saved only when model performance improves, so that storage is used efficiently and only the best models are preserved.

Acceptance Criteria

1. WHEN evaluating whether to save a checkpoint THEN the system SHALL compare current loss to the best recorded loss.
2. WHEN loss decreases by a configurable threshold THEN the system SHALL trigger checkpoint saving.
3. WHEN multiple models are trained simultaneously THEN each model SHALL have independent performance tracking.
4. WHEN checkpoint rotation occurs THEN the system SHALL keep only the best performing checkpoints.
5. WHEN performance metrics are updated THEN the system SHALL log the improvement for monitoring.
6. WHEN no improvement is detected THEN the system SHALL skip checkpoint saving to avoid unnecessary I/O.

Requirement 3: Periodic Checkpoint Saving

User Story: As a system administrator, I want checkpoints to be saved periodically regardless of performance, so that progress is preserved even during long training sessions without significant improvement.

Acceptance Criteria

1. WHEN a configurable number of training iterations have passed THEN the system SHALL save a checkpoint regardless of performance.
2. WHEN periodic saving occurs THEN the system SHALL use a separate checkpoint category to distinguish from performance-based saves.
3. WHEN the system runs for extended periods THEN periodic checkpoints SHALL ensure no more than X minutes of training progress can be lost.
4. WHEN periodic checkpoints accumulate THEN the system SHALL maintain a rolling window of recent saves.
5. WHEN storage space is limited THEN periodic checkpoints SHALL be cleaned up while preserving performance-based checkpoints.
6. WHEN the system restarts THEN it SHALL load the most recent checkpoint (either performance-based or periodic).

Requirement 4: Enhanced Training System Integration

User Story: As a developer, I want the EnhancedRealtimeTrainingSystem to properly save checkpoints, so that continuous learning progress is preserved across system restarts.

Acceptance Criteria

1. WHEN the EnhancedRealtimeTrainingSystem trains models THEN it SHALL integrate with the CheckpointManager.
2. WHEN training episodes complete THEN the system SHALL evaluate and save improved models.
3. WHEN the training system initializes THEN it SHALL load the best available checkpoints.
4. WHEN training data is collected THEN the system SHALL track performance metrics for checkpoint decisions.
5. WHEN the training system shuts down THEN it SHALL save final checkpoints before termination.
6. WHEN training resumes THEN the system SHALL continue from the last saved checkpoint state.

Requirement 5: Complete Training Data Storage

User Story: As a developer, I want complete training episodes to be stored with full input dataframes, so that training can be replayed and analyzed with all original context.

Acceptance Criteria

1. WHEN training episodes are saved THEN the system SHALL store the complete input dataframe with all model inputs (price data, indicators, market structure, etc.).
2. WHEN model actions are recorded THEN the system SHALL store the full context that led to the decision, not just the action result.
3. WHEN training cases are saved THEN they SHALL include timestamps, market conditions, and all feature vectors used by the models.
4. WHEN storing training data THEN the system SHALL preserve the exact state that can be used to reproduce the model's decision.
5. WHEN training episodes are replayed THEN the system SHALL be able to reconstruct the exact same inputs that were originally used.
6. WHEN analyzing training performance THEN complete dataframes SHALL be available for debugging and improvement.

Requirement 6: Comprehensive Performance Tracking

User Story: As a system operator, I want detailed performance metrics to be tracked and persisted, so that I can monitor training progress and model improvement over time.

Acceptance Criteria

1. WHEN models are trained THEN the system SHALL track loss values, accuracy metrics, and training timestamps.
2. WHEN performance improves THEN the system SHALL log the improvement amount and save metadata.
3. WHEN checkpoints are saved THEN the system SHALL store performance metrics alongside model weights.
4. WHEN the system starts THEN it SHALL display the performance history of loaded checkpoints.
5. WHEN multiple training sessions occur THEN the system SHALL maintain a continuous performance history.
6. WHEN performance degrades THEN the system SHALL provide alerts and revert to better checkpoints if configured.

Requirement 7: Robust Error Handling and Recovery

User Story: As a system administrator, I want checkpoint operations to be resilient to failures, so that training can continue even if individual checkpoint saves fail.

Acceptance Criteria

1. WHEN checkpoint saving fails THEN the system SHALL log the error and continue training without crashing.
2. WHEN disk space is insufficient THEN the system SHALL clean up old checkpoints and retry saving.
3. WHEN checkpoint files are corrupted THEN the system SHALL fall back to previous valid checkpoints.
4. WHEN concurrent access conflicts occur THEN the system SHALL use proper locking mechanisms.
5. WHEN the system recovers from failures THEN it SHALL validate checkpoint integrity before loading.
6. WHEN critical checkpoint operations fail repeatedly THEN the system SHALL alert administrators.

Requirement 8: Configuration and Monitoring

User Story: As a developer, I want configurable checkpoint settings and monitoring capabilities, so that I can optimize checkpoint behavior for different training scenarios.

Acceptance Criteria

1. WHEN configuring the system THEN checkpoint saving frequency SHALL be adjustable.
2. WHEN setting performance thresholds THEN the minimum improvement required for saving SHALL be configurable.
3. WHEN monitoring training THEN checkpoint save events SHALL be visible in logs and dashboards.
4. WHEN analyzing performance THEN checkpoint metadata SHALL be accessible for review.
5. WHEN tuning the system THEN checkpoint storage limits SHALL be configurable.
6. WHEN debugging issues THEN detailed checkpoint operation logs SHALL be available.

Requirement 9: Backward Compatibility and Migration

User Story: As a user, I want existing checkpoints to remain compatible, so that current model progress is preserved when the checkpoint system is enhanced.

Acceptance Criteria

1. WHEN the enhanced checkpoint system starts THEN it SHALL load existing checkpoints without issues.
2. WHEN checkpoint formats are updated THEN migration utilities SHALL convert old formats.
3. WHEN new metadata is added THEN existing checkpoints SHALL work with default values.
4. WHEN the system upgrades THEN checkpoint directories SHALL be preserved and enhanced.
5. WHEN rollback is needed THEN the system SHALL support reverting to previous checkpoint versions.
6. WHEN compatibility issues arise THEN clear error messages SHALL guide resolution.