logging channels; training steps storage

docs/LOGGING.md

@@ -0,0 +1,225 @@
+# Modular Logging System
+
+The gogo2 trading system uses a channel-based logging system that allows granular control over log verbosity for different subsystems.
+
+## Available Logging Channels
+
+| Channel | Description | Default |
+|---------|-------------|---------|
+| `core` | Core system operations | ✅ Enabled |
+| `trading` | Trading operations | ✅ Enabled |
+| `training` | Model training | ✅ Enabled |
+| `inference` | Model inference | ✅ Enabled |
+| `pivots` | Pivot calculations (Williams structure) | ❌ Disabled |
+| `data` | Data fetching/caching | ✅ Enabled |
+| `websocket` | WebSocket communications | ❌ Disabled |
+| `api` | API requests/responses | ❌ Disabled |
+| `webui` | Web UI chart requests/responses | ❌ Disabled |
+| `performance` | Performance metrics | ✅ Enabled |
+| `debug` | Debug information | ❌ Disabled |
+
+## Configuration
+
+### Method 1: Environment Variable (Recommended)
+
+Set the `LOG_CHANNELS` environment variable with a comma-separated list of channels to enable:
+
+```bash
+# Windows PowerShell
+$env:LOG_CHANNELS="core,trading,training,inference,data,performance"
+
+# Windows CMD
+set LOG_CHANNELS=core,trading,training,inference,data,performance
+
+# Linux/Mac
+export LOG_CHANNELS="core,trading,training,inference,data,performance"
+```
+
+### Method 2: Configuration File
+
+Copy `config/logging.env.example` to `config/logging.env` and edit:
+
+```bash
+LOG_CHANNELS=core,trading,training,inference,data,performance
+```
+
+Then load it before running:
+
+```bash
+# PowerShell
+Get-Content config/logging.env | ForEach-Object { 
+    if ($_ -match '^([^#][^=]+)=(.*)$') { 
+        [Environment]::SetEnvironmentVariable($matches[1], $matches[2]) 
+    } 
+}
+
+# Linux/Mac
+source config/logging.env
+```
+
+## Usage in Code
+
+### Basic Usage
+
+```python
+from utils.logging_config import get_channel_logger, LogChannel
+
+# Create a channel-specific logger
+logger = get_channel_logger(__name__, LogChannel.PIVOTS)
+
+# Log messages (only appear if PIVOTS channel is enabled)
+logger.info("Pivot point calculated")
+logger.debug("Detailed pivot data: ...")
+```
+
+### Multiple Channels
+
+```python
+# Different loggers for different purposes
+pivot_logger = get_channel_logger(__name__, LogChannel.PIVOTS)
+api_logger = get_channel_logger(__name__, LogChannel.API)
+
+pivot_logger.info("Calculating pivots...")  # Only if PIVOTS enabled
+api_logger.info("API request to Binance")   # Only if API enabled
+```
+
+### Error Logging
+
+**Important**: Errors and exceptions **always log** regardless of channel status:
+
+```python
+logger.error("This always logs!")
+logger.exception("This always logs with traceback!")
+logger.critical("This always logs!")
+```
+
+## Runtime Control
+
+### View Current Status
+
+At startup, the application prints the logging channel status:
+
+```
+=== Logging Channel Status ===
+  core            : ENABLED
+  trading         : ENABLED
+  training        : ENABLED
+  inference       : ENABLED
+  pivots          : DISABLED
+  data            : ENABLED
+  websocket       : DISABLED
+  api             : DISABLED
+  performance     : ENABLED
+  debug           : DISABLED
+===============================
+```
+
+### Interactive Control (Future Feature)
+
+```python
+from utils.logging_config import enable_channel, disable_channel
+
+# Enable a channel at runtime
+enable_channel(LogChannel.PIVOTS)
+
+# Disable a channel at runtime
+disable_channel(LogChannel.DATA)
+```
+
+## Common Scenarios
+
+### Production (Minimal Logging)
+
+```bash
+LOG_CHANNELS=core,trading,inference,performance
+```
+
+### Development (Standard)
+
+```bash
+LOG_CHANNELS=core,trading,training,inference,data,performance
+```
+
+### Debugging Pivots
+
+```bash
+LOG_CHANNELS=core,trading,training,inference,pivots,data,performance
+```
+
+### Debugging Web UI Chart Updates
+
+```bash
+LOG_CHANNELS=core,trading,training,inference,webui,data,performance
+```
+
+### Full Debug Mode
+
+```bash
+LOG_CHANNELS=core,trading,training,inference,pivots,data,websocket,api,webui,performance,debug
+```
+
+### Silent Mode (Errors Only)
+
+```bash
+LOG_CHANNELS=
+```
+
+All errors/exceptions still log, but info/debug messages are suppressed.
+
+## Log Format
+
+Channel logs include the channel name in brackets:
+
+```
+2025-11-22 12:29:17,480 - core.williams_market_structure - INFO - [pivots] Williams Market Structure initialized with 5 levels
+2025-11-22 12:29:17,489 - __main__ - INFO - [pivots] Found 4 pivot candles for ETH/USDT 1s (from 50 candles)
+2025-11-22 12:29:17,541 - __main__ - INFO - [pivots] Recalculating pivots for ETH/USDT 1m using backend data
+2025-11-22 12:44:38,871 - __main__ - INFO - [webui] Chart data request: ETH/USDT ['1s'] direction=after limit=50
+```
+
+This makes it easy to filter logs by channel using grep/findstr:
+
+```bash
+# Show only pivot logs
+grep "\[pivots\]" annotate_app.log
+
+# Show only Web UI logs
+grep "\[webui\]" annotate_app.log
+
+# Exclude pivot and webui logs (cleaner output)
+grep -v "\[pivots\]" annotate_app.log | grep -v "\[webui\]"
+```
+
+## Benefits
+
+1. **Reduced Log Noise**: Disable verbose channels in production
+2. **Targeted Debugging**: Enable only the channels you need
+3. **Performance**: Less I/O when channels are disabled
+4. **Flexibility**: Change logging at startup without code changes
+5. **Clarity**: Channel tags make logs easier to filter and understand
+
+## Migration Guide
+
+### Old Code
+
+```python
+import logging
+logger = logging.getLogger(__name__)
+
+logger.info("Pivot calculated")  # Always logs
+```
+
+### New Code
+
+```python
+from utils.logging_config import get_channel_logger, LogChannel
+
+logger = get_channel_logger(__name__, LogChannel.PIVOTS)
+
+logger.info("Pivot calculated")  # Only logs if PIVOTS channel enabled
+```
+
+### Backward Compatibility
+
+The old logging still works! Channel logging is opt-in. Only migrate code that benefits from channel filtering (e.g., verbose subsystems like pivot calculations).
+