246 lines
5.5 KiB
Markdown
246 lines
5.5 KiB
Markdown
# Auto-Load Model Configuration
|
|
|
|
The ANNOTATE application can automatically load a neural network model at startup, making it immediately available for inference and training without manual intervention.
|
|
|
|
## Configuration
|
|
|
|
### Environment Variable
|
|
|
|
Set the `AUTO_LOAD_MODEL` environment variable to specify which model to load:
|
|
|
|
```bash
|
|
# Windows PowerShell
|
|
$env:AUTO_LOAD_MODEL="Transformer"
|
|
|
|
# Windows CMD
|
|
set AUTO_LOAD_MODEL=Transformer
|
|
|
|
# Linux/Mac
|
|
export AUTO_LOAD_MODEL=Transformer
|
|
```
|
|
|
|
### Available Options
|
|
|
|
| Value | Description |
|
|
|-------|-------------|
|
|
| `Transformer` | Load the Transformer model (default) |
|
|
| `CNN` | Load the CNN model |
|
|
| `DQN` | Load the DQN agent |
|
|
| `none` | Disable auto-loading (manual load required) |
|
|
|
|
### Default Behavior
|
|
|
|
If `AUTO_LOAD_MODEL` is not set, the application defaults to loading the **Transformer** model.
|
|
|
|
## Usage Examples
|
|
|
|
### Load Transformer (Default)
|
|
|
|
```bash
|
|
# Explicitly set (same as default)
|
|
$env:AUTO_LOAD_MODEL="Transformer"
|
|
python ANNOTATE/web/app.py
|
|
```
|
|
|
|
**Output:**
|
|
```
|
|
=== Logging Channel Status ===
|
|
...
|
|
===============================
|
|
|
|
Auto-loading model: Transformer
|
|
Starting auto-load for Transformer...
|
|
Initializing TradingOrchestrator...
|
|
TradingOrchestrator initialized
|
|
Loading Transformer model...
|
|
Transformer model loaded successfully
|
|
Checkpoint: transformer_best_epoch61_20251122.pt
|
|
Accuracy: 85.67%
|
|
Loss: 0.2345
|
|
Transformer model ready for inference and training
|
|
```
|
|
|
|
### Load CNN Model
|
|
|
|
```bash
|
|
$env:AUTO_LOAD_MODEL="CNN"
|
|
python ANNOTATE/web/app.py
|
|
```
|
|
|
|
### Load DQN Agent
|
|
|
|
```bash
|
|
$env:AUTO_LOAD_MODEL="DQN"
|
|
python ANNOTATE/web/app.py
|
|
```
|
|
|
|
### Disable Auto-Loading
|
|
|
|
```bash
|
|
$env:AUTO_LOAD_MODEL="none"
|
|
python ANNOTATE/web/app.py
|
|
```
|
|
|
|
**Output:**
|
|
```
|
|
Auto-load disabled. Models available for lazy loading: DQN, CNN, Transformer
|
|
```
|
|
|
|
## Benefits
|
|
|
|
### ✅ **Immediate Availability**
|
|
- Model is ready as soon as the app starts
|
|
- No need to click "Load Models" button
|
|
- Start inference/training immediately
|
|
|
|
### ✅ **Faster Workflow**
|
|
- Skip manual model loading step
|
|
- Ideal for production/automated deployments
|
|
- Reduces startup clicks
|
|
|
|
### ✅ **Background Loading**
|
|
- Models load in a background thread
|
|
- UI remains responsive during loading
|
|
- No blocking of other operations
|
|
|
|
### ✅ **Checkpoint Auto-Discovery**
|
|
- Automatically loads the best checkpoint
|
|
- Shows checkpoint info in logs
|
|
- Displays accuracy and loss metrics
|
|
|
|
## Technical Details
|
|
|
|
### Loading Process
|
|
|
|
1. **Startup Detection**: App checks `AUTO_LOAD_MODEL` environment variable
|
|
2. **Background Thread**: Model loading happens in a separate thread
|
|
3. **Orchestrator Init**: `TradingOrchestrator` is initialized if needed
|
|
4. **Model Loading**: Specific model is loaded with best checkpoint
|
|
5. **Ready State**: Model is added to `loaded_models` dict
|
|
|
|
### Checkpoint Selection
|
|
|
|
The auto-loader uses the same checkpoint selection logic as manual loading:
|
|
|
|
1. **Database Query**: Checks `checkpoint_metadata` table for active checkpoint
|
|
2. **Filesystem Scan**: Falls back to scanning checkpoint directory
|
|
3. **Best Selection**: Chooses checkpoint with highest accuracy
|
|
4. **Metadata Display**: Shows checkpoint info in logs
|
|
|
|
### Thread Safety
|
|
|
|
- Loading happens in daemon thread
|
|
- `models_loading` flag prevents concurrent loads
|
|
- Orchestrator is thread-safe
|
|
- Training adapter is updated after load
|
|
|
|
## Integration with Other Features
|
|
|
|
### Real-Time Training
|
|
|
|
Auto-loaded models are immediately available for:
|
|
- Per-candle training
|
|
- Pivot-based training
|
|
- Live inference
|
|
|
|
### Checkpointing
|
|
|
|
Auto-loaded models use the existing checkpoint system:
|
|
- Real-time checkpoints save during training
|
|
- Best checkpoints are kept
|
|
- Metrics are tracked in database
|
|
|
|
### API Endpoints
|
|
|
|
Once auto-loaded, the model appears in:
|
|
- `/api/available-models` - Shows as loaded
|
|
- `/api/realtime-inference/start` - Ready for inference
|
|
- `/api/training/start` - Ready for training
|
|
|
|
## Configuration File
|
|
|
|
Add to your `config/logging.env` or environment:
|
|
|
|
```bash
|
|
# Model Configuration
|
|
AUTO_LOAD_MODEL=Transformer
|
|
|
|
# Logging Configuration
|
|
LOG_CHANNELS=core,trading,training,inference,data,performance
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### Model Fails to Load
|
|
|
|
**Symptoms:**
|
|
```
|
|
Error auto-loading Transformer model: ...
|
|
```
|
|
|
|
**Solutions:**
|
|
1. Check if checkpoint files exist in `models/checkpoints/`
|
|
2. Verify model architecture matches checkpoint
|
|
3. Check logs for specific error details
|
|
4. Try manual loading to see detailed error
|
|
|
|
### Wrong Model Loaded
|
|
|
|
**Symptoms:**
|
|
- Expected Transformer but CNN loaded
|
|
- Model name mismatch
|
|
|
|
**Solutions:**
|
|
1. Check `AUTO_LOAD_MODEL` environment variable
|
|
2. Restart shell to clear old environment
|
|
3. Verify spelling (case-sensitive)
|
|
|
|
### Slow Startup
|
|
|
|
**Symptoms:**
|
|
- App takes long to start
|
|
- UI unresponsive initially
|
|
|
|
**Solutions:**
|
|
- This is normal - model loading takes 5-30 seconds
|
|
- Loading happens in background, UI should still be responsive
|
|
- Check GPU availability (CUDA initialization can be slow)
|
|
- Consider disabling auto-load for development: `AUTO_LOAD_MODEL=none`
|
|
|
|
## Best Practices
|
|
|
|
### Development
|
|
|
|
```bash
|
|
# Disable auto-load for faster iteration
|
|
$env:AUTO_LOAD_MODEL="none"
|
|
```
|
|
|
|
### Production
|
|
|
|
```bash
|
|
# Auto-load Transformer for immediate availability
|
|
$env:AUTO_LOAD_MODEL="Transformer"
|
|
```
|
|
|
|
### Testing
|
|
|
|
```bash
|
|
# Load specific model for testing
|
|
$env:AUTO_LOAD_MODEL="DQN"
|
|
```
|
|
|
|
### CI/CD
|
|
|
|
```bash
|
|
# Disable in CI pipelines (no GPU)
|
|
export AUTO_LOAD_MODEL=none
|
|
```
|
|
|
|
## See Also
|
|
|
|
- [Logging Configuration](LOGGING.md) - Configure logging channels
|
|
- [Checkpoint System](../ANNOTATE/core/real_training_adapter.py) - Checkpoint management
|
|
- [Model Training](../NN/models/) - Model architectures
|
|
|