Vorndran, M.; Schütz, A.; Bendix, J. & Thies, B. (2021-11-05). <b>Training and validation weaknesses in pointwise classification-based radiation fog forecast using machine learning algorithms </b>. Presented at AK Klima, Passau.
Resource Description
Title:
Training and validation weaknesses in pointwise classification-based radiation fog forecast using machine learning algorithms
FOR816dw ID:
507
Publication Date:
2021-11-06
License and Usage Rights:
Resource Owner(s):
Individual:
Michaela Vorndran
Contact:
email:
michaela.schuetz <at> geo.uni-marburg.de
Germany
Individual:
Adrian Schütz
Contact:
email:
webmaster <at> lcrs.de
Individual:
Jörg Bendix
Contact:
email:
bendix <at> staff.uni-marburg.de
Deutschhausstraße 12
Room No. 02 A 48
35032 Marburg
Faculty of Geography
Germany
Fog forecasting still shows large inaccuracies in accurately predicting fog formation, dissipation and duration. Since a few years, Machine learning (ML) algorithms are increasingly used in addition to numerical fog forecasts because of their computational speed and ability to learn non-linear interactions between the variables. Due to their black-box nature, precise and accurate training and evaluation is vital to prevent insufficient training or meaningless scores. Three main points important for fog prediction are explained in the following. <br/>
1. Fog forecasting datasets consist of autocorrelated variables. In most cases, there is an information leakage between the training and test data sets which are used to evaluate the model performance. This information leakage can have an impact on the performance scores because the stronger the information flow, the easier it is for the model to memorize. <br/>
2. Fog forecasting datasets have a temporal order. To be able to make statements about the performance of an operational model this temporal order should already be simulated during model training and evaluation. This is because for an operational model, the training data points are always older than the data points to be predicted. Commonly used training methods neglect this fact. <br/>
3. Time series used for fog forecasting usually have a large imbalance between the frequency of the fog class and non-fog class. This imbalance can have an unfavorable interaction with the confusion matrix based meteorological scores that are widely used for evaluation. All of the aforementioned points, if not considered, can lead to an insufficient forecast without even being noticed. <br/>
Therefore, the negative influence on the model score of two commonly used training methods that neglect the points named above will be shown using an XGBoost model and a logistic regression model. In comparison, a training and evaluation method was evaluated that maintains the temporal order and thus simulates the performance of an operational model. It will also be shown that common meteorological scores, since they are computed based on a confusion matrix, share a weakness when the data set is unbalanced: Persistence behavior remains undetected. <br/>
The study is funded by the DFG research project “FOG-ML FOrecasting radiation foG by combining station and satellite data using Machine Learning”.
Keywords:
| fog forecasting | station data | Machine learning | Decision Trees | Classification | XGBoost |
