reduce logging. actual training

ANNOTATE/LOGGING_CONFIGURATION.md

@@ -0,0 +1,184 @@
+# Logging Configuration
+
+## Issue: Excessive Werkzeug Logs
+
+### Problem
+```
+2025-10-31 03:23:53,478 - werkzeug - INFO - 127.0.0.1 - - [31/Oct/2025 03:23:53] "POST /api/training-progress HTTP/1.1" 200 -
+2025-10-31 03:23:55,519 - werkzeug - INFO - 127.0.0.1 - - [31/Oct/2025 03:23:55] "POST /api/training-progress HTTP/1.1" 200 -
+2025-10-31 03:23:56,533 - werkzeug - INFO - 127.0.0.1 - - [31/Oct/2025 03:23:56] "POST /api/training-progress HTTP/1.1" 200 -
+...
+```
+
+**Cause**: The frontend polls `/api/training-progress` every 1-2 seconds, and Flask's werkzeug logger logs every request at INFO level.
+
+---
+
+## Solution
+
+### Fixed in `ANNOTATE/web/app.py`
+
+```python
+# Initialize Flask app
+self.server = Flask(
+    __name__,
+    template_folder='templates',
+    static_folder='static'
+)
+
+# Suppress werkzeug request logs (reduce noise from polling endpoints)
+werkzeug_logger = logging.getLogger('werkzeug')
+werkzeug_logger.setLevel(logging.WARNING)  # Only show warnings and errors, not INFO
+```
+
+**Result**: Werkzeug will now only log warnings and errors, not every request.
+
+---
+
+## Logging Levels
+
+### Before (Noisy)
+```
+INFO - Every request logged
+INFO - GET /api/chart-data
+INFO - POST /api/training-progress
+INFO - GET /static/css/style.css
+... (hundreds of lines per minute)
+```
+
+### After (Clean)
+```
+WARNING - Only important events
+ERROR - Only errors
+... (quiet unless something is wrong)
+```
+
+---
+
+## Customization
+
+### Show Only Errors
+```python
+werkzeug_logger.setLevel(logging.ERROR)  # Only errors
+```
+
+### Show All Requests (Debug Mode)
+```python
+werkzeug_logger.setLevel(logging.INFO)  # All requests (default)
+```
+
+### Selective Filtering
+```python
+# Custom filter to exclude specific endpoints
+class ExcludeEndpointFilter(logging.Filter):
+    def filter(self, record):
+        # Exclude training-progress endpoint
+        return '/api/training-progress' not in record.getMessage()
+
+werkzeug_logger.addFilter(ExcludeEndpointFilter())
+```
+
+---
+
+## Other Loggers
+
+### Application Logger
+```python
+# Your application logs (keep at INFO)
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.INFO)
+```
+
+### Third-Party Libraries
+```python
+# Suppress noisy third-party loggers
+logging.getLogger('urllib3').setLevel(logging.WARNING)
+logging.getLogger('requests').setLevel(logging.WARNING)
+logging.getLogger('matplotlib').setLevel(logging.WARNING)
+```
+
+---
+
+## Log File Configuration
+
+### Current Setup
+```python
+log_file = Path(__file__).parent.parent / 'logs' / f'annotate_{datetime.now().strftime("%Y%m%d")}.log'
+
+logging.basicConfig(
+    level=logging.INFO,
+    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
+    handlers=[
+        logging.FileHandler(log_file),
+        logging.StreamHandler(sys.stdout)
+    ]
+)
+```
+
+### Recommended: Separate Log Files
+```python
+# Application logs
+app_log = 'logs/annotate_app.log'
+app_handler = logging.FileHandler(app_log)
+app_handler.setLevel(logging.INFO)
+
+# Request logs (if needed)
+request_log = 'logs/annotate_requests.log'
+request_handler = logging.FileHandler(request_log)
+request_handler.setLevel(logging.DEBUG)
+
+# Configure werkzeug to use separate file
+werkzeug_logger = logging.getLogger('werkzeug')
+werkzeug_logger.addHandler(request_handler)
+werkzeug_logger.setLevel(logging.WARNING)  # Still suppress in main log
+```
+
+---
+
+## Summary
+
+### What Changed
+- ✅ Werkzeug logger set to WARNING level
+- ✅ No more INFO logs for every request
+- ✅ Still logs errors and warnings
+- ✅ Application logs unchanged
+
+### Result
+```
+Before: 100+ log lines per minute (polling)
+After: 0-5 log lines per minute (only important events)
+```
+
+### To Revert
+```python
+# Show all requests again
+werkzeug_logger.setLevel(logging.INFO)
+```
+
+---
+
+## Best Practices
+
+1. **Production**: Use WARNING or ERROR for werkzeug
+2. **Development**: Use INFO for debugging
+3. **Polling Endpoints**: Always suppress or use separate log file
+4. **Application Logs**: Keep at INFO or DEBUG as needed
+5. **Third-Party**: Suppress noisy libraries
+
+---
+
+## Testing
+
+After the change, you should see:
+```
+✅ No more werkzeug INFO logs
+✅ Application logs still visible
+✅ Errors still logged
+✅ Clean console output
+```
+
+If you need to see requests for debugging:
+```python
+# Temporarily enable
+logging.getLogger('werkzeug').setLevel(logging.INFO)
+```