38 lines
3.9 KiB
Markdown
38 lines
3.9 KiB
Markdown
I. our system architecture is such that we have data inflow with different rates from different providers. our data flow though the system should be single and centralized. I think our orchestrator class is taking that role. since our different data feeds have different rates (and also each model has different inference times and cycle) our orchestrator should keep cache of the latest available data and keep track of the rates and statistics of each data source - being data api or our own model outputs. so the available data is constantly updated and refreshed in realtime by multiple sources, and is also consumed by all smodels
|
|
II. orchestrator should also be responsible for the data ingestion and processing. it should be able to handle the data from different sources and process them in a unified way. it may hold cache of the latest available data and keep track of the rates and statistics of each data source - being data api or our own model outputs. so the available data is constantly updated and refreshed in realtime by multiple sources, and is also consumed by all smodels. orchestrator holds business logic and rules, but also uses our special decision model which is at the end of the data flow and is used to lean the effectivenes of the other model outputs in contribute to succeessful prediction. this way we will have learned signal weight. it should be trained on each price prediction data point and each trade signal data point.
|
|
orchestrator can use the various trainer classes as different models have different training requirements and pipelines.
|
|
|
|
III. models we currently use (architecture is expandable with easy adaption to new models)
|
|
- cnn price prediction model - uses calculated multilevel pivot points and historical price data to predict the next pivot point for each level.
|
|
- DQN RL model outputs trade signals
|
|
- transformer model outputs price prediction
|
|
- COB RL model outputs trade signals - it is trained on cob (cached all COB data for period of time not just current order book. it should be a 2d matrix 1s aggregated ) and some indicators cummulative cob imbalance for different timeframes. we get COB snapshots every couple hundred miliseconds and we cache and aggregate them to have a COB history. 1d matrix from the API to 2d amtrix as model inputs. as both raw ticks and 1s averaged.
|
|
- decision model - it is trained on price prediction and trade signals to learn the effectiveness of the other models in contribute to succeessful prediction. outputs the final trade signal.
|
|
|
|
|
|
IV. by default all models take full current data frames available in the orchestrator on inference as base data - different aspects of the data are updated at different rates. main data frame includes 5 price charts
|
|
class UniversalDataAdapter:
|
|
- 1s 1m 1h ETH charts and ETH and BTC ticks. orchestrator can use and extend the UniversalDataAdapter class to add new data sources and data types.
|
|
- - cob models are different and they get fast realtime raw dob data ticks and should be agile to inference and procude outputs but yet able to learn.
|
|
|
|
V. Training and hardware.
|
|
- we should load the models in a way that we do a back propagation and other model specificic training at realtime as training examples emerge from the realtime data we process. we will save only the best examples (the realtime data dumps we feed to the models) so we can cold start other models if we change the architecture. i
|
|
- we use GPU if available for training and inference for optimised performance.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
dashboard should be able to show the data from the orchestrator and hold some amount of bussiness logic related to UI representations, but limited. it mainly relies on the orchestrator to provide the data and the models to make the decisions. dash's main job is to show the data and the models' decisions in a user friendly way.
|
|
|
|
|
|
|
|
ToDo:
|
|
check and integrade EnhancedRealtimeTrainingSystem and EnhancedRLTrainingIntegrator into orchestrator
|
|
|
|
|
|
|
|
|