diff --git a/ANNOTATE/ANNOTATION_WORKFLOW.md b/ANNOTATE/ANNOTATION_WORKFLOW.md new file mode 100644 index 0000000..e69de29 diff --git a/ANNOTATE/USAGE_GUIDE.md b/ANNOTATE/USAGE_GUIDE.md new file mode 100644 index 0000000..12f0029 --- /dev/null +++ b/ANNOTATE/USAGE_GUIDE.md @@ -0,0 +1,306 @@ +# ANNOTATE - Usage Guide + +## 🎯 Quick Start + +### Starting the Application +```bash +python ANNOTATE/web/app.py +``` +Access at: **http://127.0.0.1:8051** + +--- + +## 📊 Creating Annotations + +### Method 1: Click to Mark (Recommended) +1. **Navigate** to the time period you want to annotate +2. **Click on the chart** at the entry point + - You'll see "Entry marked" status + - A temporary marker appears +3. **Click again** at the exit point + - Annotation is saved automatically + - Visual markers appear: ▲ (entry) and ▼ (exit) + - P&L percentage is calculated and displayed + +### What Gets Captured +When you create an annotation, the system captures: +- ✅ **Entry timestamp and price** +- ✅ **Exit timestamp and price** +- ✅ **Full market state** (OHLCV for all 4 timeframes) +- ✅ **Direction** (LONG/SHORT) +- ✅ **P&L percentage** +- ✅ **Market context** at both entry and exit + +This ensures the annotation contains **exactly the same data** your models will see during training! + +--- + +## ✏️ Editing Annotations + +### Method 1: Click on P&L Label +1. **Click the P&L label** (the percentage with 🗑️ icon) +2. Choose action: + - **1** - Move entry point + - **2** - Move exit point + - **3** - Delete annotation + +### Method 2: From Sidebar +1. Find annotation in the right sidebar +2. Click the **eye icon** (👁️) to view +3. Click the **trash icon** (🗑️) to delete + +### Moving Entry/Exit Points +1. Click on annotation → Choose "1" or "2" +2. The current point is removed +3. The other point stays as reference (grayed out) +4. **Click on chart** to set new position +5. Annotation is updated automatically + +--- + +## 🎨 Visual Indicators + +### On Charts +- **▲ Green Triangle** = LONG entry point +- **▲ Red Triangle** = SHORT entry point +- **▼ Green Triangle** = LONG exit point +- **▼ Red Triangle** = SHORT exit point +- **Dashed Line** = Connects entry to exit +- **P&L Label** = Shows profit/loss percentage +- **🗑️ Icon** = Click to edit/delete + +### Color Coding +- **Green** = LONG trade (buy low, sell high) +- **Red** = SHORT trade (sell high, buy low) +- **Positive P&L** = Green text +- **Negative P&L** = Red text + +--- + +## 🗂️ Managing Annotations + +### Viewing All Annotations +- Right sidebar shows all annotations +- Sorted by creation time +- Shows: Direction, Timeframe, P&L, Timestamp + +### Filtering +- Annotations are grouped by symbol +- Switch symbols using the dropdown + +### Exporting +1. Click **download button** at top of annotation list +2. All annotations exported to JSON file +3. File includes full market context + +--- + +## 📦 Generating Test Cases + +### Automatic Generation +When you save an annotation, the system: +1. ✅ Captures market state at entry time +2. ✅ Captures market state at exit time +3. ✅ Stores OHLCV data for all timeframes +4. ✅ Calculates expected outcome (P&L, direction) + +### Manual Generation +1. Find annotation in sidebar +2. Click **file icon** (📄) +3. Test case generated and saved to: + ``` + ANNOTATE/data/test_cases/annotation_.json + ``` + +### Test Case Format +```json +{ + "test_case_id": "annotation_uuid", + "symbol": "ETH/USDT", + "timestamp": "2024-01-15T10:30:00Z", + "action": "BUY", + "market_state": { + "ohlcv_1s": { /* 100 candles */ }, + "ohlcv_1m": { /* 100 candles */ }, + "ohlcv_1h": { /* 100 candles */ }, + "ohlcv_1d": { /* 100 candles */ } + }, + "expected_outcome": { + "direction": "LONG", + "profit_loss_pct": 2.5, + "entry_price": 2400.50, + "exit_price": 2460.75 + } +} +``` + +--- + +## ⌨️ Keyboard Shortcuts + +- **← Left Arrow** = Navigate backward in time +- **→ Right Arrow** = Navigate forward in time +- **Space** = Mark point (when chart focused) +- **Esc** = Cancel pending annotation + +--- + +## 🎯 Best Practices + +### 1. Be Selective +- Only mark **clear, high-confidence** trades +- Quality > Quantity +- Look for obvious patterns + +### 2. Use Multiple Timeframes +- Check all 4 timeframes before marking +- Confirm pattern across timeframes +- Look for confluence + +### 3. Document Your Reasoning +- Add notes to annotations (future feature) +- Explain why you marked the trade +- Note key indicators or patterns + +### 4. Review Before Generating +- Verify entry/exit points are correct +- Check P&L calculation makes sense +- Ensure market context is complete + +### 5. Organize by Strategy +- Group similar trade types +- Use consistent marking criteria +- Build a library of patterns + +--- + +## 🔧 Troubleshooting + +### Clicks Not Working +- **Issue**: Chart clicks don't register +- **Solution**: + - Make sure you're clicking on the **candlestick** (not volume bars) + - Click on the **body or wick** of a candle + - Avoid clicking on empty space + +### Annotations Not Appearing +- **Issue**: Saved annotations don't show on charts +- **Solution**: + - Refresh the page + - Check the correct symbol is selected + - Verify annotation is for the visible timeframe + +### Can't Edit Annotation +- **Issue**: Edit mode not working +- **Solution**: + - Click directly on the **P&L label** (percentage text) + - Or use the sidebar icons + - Make sure annotation mode is enabled + +### Market Context Missing +- **Issue**: Test case has empty market_state +- **Solution**: + - Ensure DataProvider has cached data + - Check timestamp is within available data range + - Verify all timeframes have data + +--- + +## 💡 Tips & Tricks + +### Tip 1: Quick Navigation +Use the **quick range buttons** (1h, 4h, 1d, 1w) to jump to different time periods quickly. + +### Tip 2: Zoom for Precision +- **Scroll wheel** to zoom in/out +- **Drag** to pan +- Get precise entry/exit points + +### Tip 3: Check All Timeframes +Before marking, scroll through all 4 charts to confirm the pattern is valid across timeframes. + +### Tip 4: Start with Recent Data +Begin annotating recent data where you remember the market conditions clearly. + +### Tip 5: Batch Export +Export annotations regularly to backup your work. + +--- + +## 📊 Data Consistency + +### Why It Matters +The annotation system uses the **same DataProvider** as your training and inference systems. This means: + +✅ **Same data source** +✅ **Same data quality** +✅ **Same data structure** +✅ **Same timeframes** +✅ **Same caching** + +### What This Guarantees +When you train a model on annotated data: +- The model sees **exactly** what you saw +- No data discrepancies +- No format mismatches +- Perfect consistency + +--- + +## 🎓 Example Workflow + +### Scenario: Mark a Breakout Trade + +1. **Navigate** to ETH/USDT on 2024-01-15 +2. **Identify** a breakout pattern on 1m chart +3. **Confirm** on 1h chart (uptrend) +4. **Click** at breakout point: $2400.50 (10:30:00) +5. **Click** at target: $2460.75 (10:35:00) +6. **Result**: LONG trade, +2.51% P&L +7. **Verify**: Check all timeframes show the pattern +8. **Generate**: Click file icon to create test case +9. **Train**: Use test case to train model + +--- + +## 📝 Storage Locations + +### Annotations +``` +ANNOTATE/data/annotations/annotations_db.json +``` + +### Test Cases +``` +ANNOTATE/data/test_cases/annotation_.json +``` + +### Exports +``` +ANNOTATE/data/annotations/export_.json +``` + +--- + +## 🚀 Next Steps + +After creating annotations: + +1. **Generate test cases** for all annotations +2. **Review test cases** to verify market context +3. **Train models** using the test cases +4. **Evaluate performance** with inference simulation +5. **Iterate** - mark more trades, refine patterns + +--- + +## 📞 Support + +For issues or questions: +- Check `ANNOTATE/README.md` for technical details +- Review `ANNOTATE/IMPLEMENTATION_SUMMARY.md` for architecture +- See `ANNOTATE/STATUS.md` for current status + +--- + +**Happy Annotating!** 🎉 diff --git a/ANNOTATE/core/annotation_manager.py b/ANNOTATE/core/annotation_manager.py index 6aa8fac..abe2fab 100644 --- a/ANNOTATE/core/annotation_manager.py +++ b/ANNOTATE/core/annotation_manager.py @@ -99,6 +99,12 @@ class AnnotationManager: direction = 'SHORT' profit_loss_pct = ((entry_price - exit_price) / entry_price) * 100 + # Store complete market context for training + market_context = { + 'entry_state': entry_market_state or {}, + 'exit_state': exit_market_state or {} + } + annotation = TradeAnnotation( annotation_id=str(uuid.uuid4()), symbol=symbol, @@ -106,10 +112,13 @@ class AnnotationManager: entry=entry_point, exit=exit_point, direction=direction, - profit_loss_pct=profit_loss_pct + profit_loss_pct=profit_loss_pct, + market_context=market_context ) logger.info(f"Created annotation: {annotation.annotation_id} ({direction}, {profit_loss_pct:.2f}%)") + logger.info(f" Entry state: {len(entry_market_state or {})} timeframes") + logger.info(f" Exit state: {len(exit_market_state or {})} timeframes") return annotation def save_annotation(self, annotation: TradeAnnotation): diff --git a/ANNOTATE/web/app.py b/ANNOTATE/web/app.py index e6e5289..d84b423 100644 --- a/ANNOTATE/web/app.py +++ b/ANNOTATE/web/app.py @@ -138,17 +138,39 @@ class AnnotationDashboard: @self.server.route('/') def index(): - """Main dashboard page""" + """Main dashboard page - loads existing annotations""" try: - # Get current annotations + # Get all existing annotations annotations = self.annotation_manager.get_annotations() + # Convert to serializable format + annotations_data = [] + for ann in annotations: + if hasattr(ann, '__dict__'): + ann_dict = ann.__dict__ + else: + ann_dict = ann + + # Ensure all fields are JSON serializable + annotations_data.append({ + 'annotation_id': ann_dict.get('annotation_id'), + 'symbol': ann_dict.get('symbol'), + 'timeframe': ann_dict.get('timeframe'), + 'entry': ann_dict.get('entry'), + 'exit': ann_dict.get('exit'), + 'direction': ann_dict.get('direction'), + 'profit_loss_pct': ann_dict.get('profit_loss_pct'), + 'notes': ann_dict.get('notes', ''), + 'created_at': ann_dict.get('created_at') + }) + + logger.info(f"Loading dashboard with {len(annotations_data)} existing annotations") + # Prepare template data template_data = { 'current_symbol': 'ETH/USDT', 'timeframes': ['1s', '1m', '1h', '1d'], - 'annotations': [ann.__dict__ if hasattr(ann, '__dict__') else ann - for ann in annotations] + 'annotations': annotations_data } return render_template('annotation_dashboard.html', **template_data) @@ -261,16 +283,72 @@ class AnnotationDashboard: @self.server.route('/api/save-annotation', methods=['POST']) def save_annotation(): - """Save a new annotation""" + """Save a new annotation with full market context""" try: data = request.get_json() - # Create annotation + # Capture market state at entry and exit times + entry_market_state = {} + exit_market_state = {} + + if self.data_loader: + try: + # Parse timestamps + entry_time = datetime.fromisoformat(data['entry']['timestamp'].replace('Z', '+00:00')) + exit_time = datetime.fromisoformat(data['exit']['timestamp'].replace('Z', '+00:00')) + + # Fetch market data for all timeframes at entry time + timeframes = ['1s', '1m', '1h', '1d'] + for tf in timeframes: + df = self.data_loader.get_data( + symbol=data['symbol'], + timeframe=tf, + end_time=entry_time, + limit=100 + ) + + if df is not None and not df.empty: + entry_market_state[f'ohlcv_{tf}'] = { + 'timestamps': df.index.strftime('%Y-%m-%d %H:%M:%S').tolist(), + 'open': df['open'].tolist(), + 'high': df['high'].tolist(), + 'low': df['low'].tolist(), + 'close': df['close'].tolist(), + 'volume': df['volume'].tolist() + } + + # Fetch market data at exit time + for tf in timeframes: + df = self.data_loader.get_data( + symbol=data['symbol'], + timeframe=tf, + end_time=exit_time, + limit=100 + ) + + if df is not None and not df.empty: + exit_market_state[f'ohlcv_{tf}'] = { + 'timestamps': df.index.strftime('%Y-%m-%d %H:%M:%S').tolist(), + 'open': df['open'].tolist(), + 'high': df['high'].tolist(), + 'low': df['low'].tolist(), + 'close': df['close'].tolist(), + 'volume': df['volume'].tolist() + } + + logger.info(f"Captured market state: {len(entry_market_state)} timeframes at entry, {len(exit_market_state)} at exit") + + except Exception as e: + logger.error(f"Error capturing market state: {e}") + + # Create annotation with market context annotation = self.annotation_manager.create_annotation( entry_point=data['entry'], exit_point=data['exit'], symbol=data['symbol'], - timeframe=data['timeframe'] + timeframe=data['timeframe'], + entry_market_state=entry_market_state, + exit_market_state=exit_market_state ) # Save annotation diff --git a/ANNOTATE/web/static/js/annotation_manager.js b/ANNOTATE/web/static/js/annotation_manager.js index f869e70..fda3169 100644 --- a/ANNOTATE/web/static/js/annotation_manager.js +++ b/ANNOTATE/web/static/js/annotation_manager.js @@ -6,13 +6,14 @@ class AnnotationManager { constructor(chartManager) { this.chartManager = chartManager; this.pendingAnnotation = null; + this.editingAnnotation = null; this.enabled = true; console.log('AnnotationManager initialized'); } /** - * Handle chart click for marking entry/exit + * Handle chart click for marking entry/exit or editing */ handleChartClick(clickData) { if (!this.enabled) { @@ -20,6 +21,12 @@ class AnnotationManager { return; } + // Check if we're editing an existing annotation + if (this.editingAnnotation) { + this.handleEditClick(clickData); + return; + } + if (!this.pendingAnnotation) { // Mark entry point this.markEntry(clickData); @@ -29,6 +36,95 @@ class AnnotationManager { } } + /** + * Handle click while editing an annotation + */ + handleEditClick(clickData) { + const editing = this.editingAnnotation; + const original = editing.original; + + if (editing.editMode === 'entry') { + // Update entry point + const newAnnotation = { + ...original, + entry: { + timestamp: clickData.timestamp, + price: clickData.price, + index: clickData.index + } + }; + + // Recalculate P&L + const entryPrice = newAnnotation.entry.price; + const exitPrice = newAnnotation.exit.price; + newAnnotation.direction = exitPrice > entryPrice ? 'LONG' : 'SHORT'; + newAnnotation.profit_loss_pct = ((exitPrice - entryPrice) / entryPrice) * 100; + + // Delete old annotation and save new one + this.deleteAndSaveAnnotation(editing.annotation_id, newAnnotation); + + } else if (editing.editMode === 'exit') { + // Update exit point + const newAnnotation = { + ...original, + exit: { + timestamp: clickData.timestamp, + price: clickData.price, + index: clickData.index + } + }; + + // Recalculate P&L + const entryPrice = newAnnotation.entry.price; + const exitPrice = newAnnotation.exit.price; + newAnnotation.direction = exitPrice > entryPrice ? 'LONG' : 'SHORT'; + newAnnotation.profit_loss_pct = ((exitPrice - entryPrice) / entryPrice) * 100; + + // Delete old annotation and save new one + this.deleteAndSaveAnnotation(editing.annotation_id, newAnnotation); + } + + // Clear editing mode + this.editingAnnotation = null; + window.showSuccess('Annotation updated'); + } + + /** + * Delete old annotation and save updated one + */ + deleteAndSaveAnnotation(oldId, newAnnotation) { + // Delete old + fetch('/api/delete-annotation', { + method: 'POST', + headers: {'Content-Type': 'application/json'}, + body: JSON.stringify({annotation_id: oldId}) + }) + .then(() => { + // Save new + return fetch('/api/save-annotation', { + method: 'POST', + headers: {'Content-Type': 'application/json'}, + body: JSON.stringify(newAnnotation) + }); + }) + .then(response => response.json()) + .then(data => { + if (data.success) { + // Update app state + window.appState.annotations = window.appState.annotations.filter(a => a.annotation_id !== oldId); + window.appState.annotations.push(data.annotation); + + // Update UI + window.renderAnnotationsList(window.appState.annotations); + this.chartManager.removeAnnotation(oldId); + this.chartManager.addAnnotation(data.annotation); + } + }) + .catch(error => { + window.showError('Failed to update annotation: ' + error.message); + }); + } + /** * Mark entry point */ diff --git a/ANNOTATE/web/static/js/chart_manager.js b/ANNOTATE/web/static/js/chart_manager.js index b7262aa..0426c9a 100644 --- a/ANNOTATE/web/static/js/chart_manager.js +++ b/ANNOTATE/web/static/js/chart_manager.js @@ -139,11 +139,27 @@ class ChartManager { annotations: [] }; - // Add click handler for annotations + // Add click handler for chart plotElement.on('plotly_click', (eventData) => { this.handleChartClick(timeframe, eventData); }); + // Add click handler for annotations + plotElement.on('plotly_clickannotation', (eventData) => { + const annotationName = eventData.annotation.name; + if (annotationName) { + const parts = annotationName.split('_'); + const action = parts[0]; // 'entry', 'exit', or 'delete' + const annotationId = parts[1]; + + if (action === 'delete') { + this.handleAnnotationClick(annotationId, 'delete'); + } else { + this.handleAnnotationClick(annotationId, 'edit'); + } + } + }); + // Add hover handler to update info plotElement.on('plotly_hover', (eventData) => { this.updateChartInfo(timeframe, eventData); @@ -159,10 +175,23 @@ class ChartManager { if (!eventData.points || eventData.points.length === 0) return; const point = eventData.points[0]; + + // Get the actual price from candlestick data + let price; + if (point.data.type === 'candlestick') { + // For candlestick, use close price + price = point.data.close[point.pointIndex]; + } else if (point.data.type === 'bar') { + // Skip volume bar clicks + return; + } else { + price = point.y; + } + const clickData = { timeframe: timeframe, timestamp: point.x, - price: point.close || point.y, + price: price, index: point.pointIndex }; @@ -255,7 +284,7 @@ class ChartManager { const entryPrice = ann.entry.price; const exitPrice = ann.exit.price; - // Entry marker + // Entry marker (clickable) plotlyAnnotations.push({ x: entryTime, y: entryPrice, @@ -266,10 +295,12 @@ class ChartManager { color: ann.direction === 'LONG' ? '#10b981' : '#ef4444' }, xanchor: 'center', - yanchor: 'bottom' + yanchor: 'bottom', + captureevents: true, + name: `entry_${ann.annotation_id}` }); - // Exit marker + // Exit marker (clickable) plotlyAnnotations.push({ x: exitTime, y: exitPrice, @@ -280,10 +311,12 @@ class ChartManager { color: ann.direction === 'LONG' ? '#10b981' : '#ef4444' }, xanchor: 'center', - yanchor: 'top' + yanchor: 'top', + captureevents: true, + name: `exit_${ann.annotation_id}` }); - // P&L label + // P&L label with delete button const midTime = new Date((new Date(entryTime).getTime() + new Date(exitTime).getTime()) / 2); const midPrice = (entryPrice + exitPrice) / 2; const pnlColor = ann.profit_loss_pct >= 0 ? '#10b981' : '#ef4444'; @@ -291,7 +324,7 @@ class ChartManager { plotlyAnnotations.push({ x: midTime, y: midPrice, - text: `${ann.profit_loss_pct >= 0 ? '+' : ''}${ann.profit_loss_pct.toFixed(2)}%`, + text: `${ann.profit_loss_pct >= 0 ? '+' : ''}${ann.profit_loss_pct.toFixed(2)}% 🗑️`, showarrow: true, arrowhead: 0, ax: 0, @@ -304,10 +337,12 @@ class ChartManager { bgcolor: '#1f2937', bordercolor: pnlColor, borderwidth: 1, - borderpad: 4 + borderpad: 4, + captureevents: true, + name: `delete_${ann.annotation_id}` }); - // Connecting line + // Connecting line (clickable for selection) plotlyShapes.push({ type: 'line', x0: entryTime, @@ -318,7 +353,8 @@ class ChartManager { color: ann.direction === 'LONG' ? '#10b981' : '#ef4444', width: 2, dash: 'dash' - } + }, + name: `line_${ann.annotation_id}` }); }); @@ -331,6 +367,25 @@ class ChartManager { console.log(`Updated ${timeframeAnnotations.length} annotations for ${timeframe}`); } + /** + * Handle annotation click for editing/deleting + */ + handleAnnotationClick(annotationId, action) { + console.log(`Annotation ${action}:`, annotationId); + + if (action === 'delete') { + if (confirm('Delete this annotation?')) { + if (window.deleteAnnotation) { + window.deleteAnnotation(annotationId); + } + } + } else if (action === 'edit') { + if (window.appState && window.appState.chartManager) { + window.appState.chartManager.editAnnotation(annotationId); + } + } + } + /** * Highlight annotation */ @@ -366,27 +421,88 @@ class ChartManager { } /** - * Edit annotation + * Edit annotation - allows moving entry/exit points */ editAnnotation(annotationId) { const annotation = this.annotations[annotationId]; if (!annotation) return; - // Remove from charts - this.removeAnnotation(annotationId); + // Show edit dialog + const action = prompt( + 'Edit annotation:\n' + + '1 - Move entry point\n' + + '2 - Move exit point\n' + + '3 - Delete annotation\n' + + 'Enter choice (1-3):', + '1' + ); - // Set as pending annotation for editing - if (window.appState && window.appState.annotationManager) { - window.appState.annotationManager.pendingAnnotation = { - annotation_id: annotationId, - symbol: annotation.symbol, - timeframe: annotation.timeframe, - entry: annotation.entry, - isEditing: true - }; + if (action === '1') { + // Move entry point + window.showSuccess('Click on chart to set new entry point'); - document.getElementById('pending-annotation-status').style.display = 'block'; + // Store annotation for editing + if (window.appState && window.appState.annotationManager) { + window.appState.annotationManager.editingAnnotation = { + annotation_id: annotationId, + original: annotation, + editMode: 'entry' + }; + + // Remove current annotation from display + this.removeAnnotation(annotationId); + + // Show exit marker as reference + const chart = this.charts[annotation.timeframe]; + if (chart) { + Plotly.relayout(chart.plotId, { + annotations: [{ + x: annotation.exit.timestamp, + y: annotation.exit.price, + text: '▼ (exit)', + showarrow: true, + arrowhead: 2, + ax: 0, + ay: 40, + font: {size: 14, color: '#9ca3af'} + }] + }); + } + } + } else if (action === '2') { + // Move exit point window.showSuccess('Click on chart to set new exit point'); + + if (window.appState && window.appState.annotationManager) { + window.appState.annotationManager.editingAnnotation = { + annotation_id: annotationId, + original: annotation, + editMode: 'exit' + }; + + // Remove current annotation from display + this.removeAnnotation(annotationId); + + // Show entry marker as reference + const chart = this.charts[annotation.timeframe]; + if (chart) { + Plotly.relayout(chart.plotId, { + annotations: [{ + x: annotation.entry.timestamp, + y: annotation.entry.price, + text: '▲ (entry)', + showarrow: true, + arrowhead: 2, + ax: 0, + ay: -40, + font: {size: 14, color: '#9ca3af'} + }] + }); + } + } + } else if (action === '3') { + // Delete + this.handleAnnotationClick(annotationId, 'delete'); } } diff --git a/ANNOTATE/web/templates/annotation_dashboard.html b/ANNOTATE/web/templates/annotation_dashboard.html index 3de6efe..d87d474 100644 --- a/ANNOTATE/web/templates/annotation_dashboard.html +++ b/ANNOTATE/web/templates/annotation_dashboard.html @@ -43,7 +43,7 @@ {% block extra_js %} {% endblock %}