Applying machine learning for drought prediction in a perfect model framework using data from a large ensemble of climate simulations

Felsche, Elizaveta; Ludwig, Ralf

doi:https://doi.org/10.5194/nhess-21-3679-2021

Articles | Volume 21, issue 12

https://doi.org/10.5194/nhess-21-3679-2021

© Author(s) 2021. This work is distributed under
the Creative Commons Attribution 4.0 License.

Special issue:

Recent advances in drought and water scarcity monitoring,...

https://doi.org/10.5194/nhess-21-3679-2021

© Author(s) 2021. This work is distributed under
the Creative Commons Attribution 4.0 License.

Articles | Volume 21, issue 12

Research article

|

03 Dec 2021

Research article |

| 03 Dec 2021

Applying machine learning for drought prediction in a perfect model framework using data from a large ensemble of climate simulations

Elizaveta Felsche and Ralf Ludwig

Download

Final revised paper (published on 03 Dec 2021)
Preprint (discussion started on 15 Apr 2021)

Interactive discussion

Status: closed

RC1:
'Comment on nhess-2021-110', Anonymous Referee #1, 10 Jun 2021

As presented by authors this study consists of two parts:

the first part focuses on a systematic search for the best performing setup of ANN models for Munich and Lisbon.

the second part focuses on the analysis the best performing models using explainable AI methods.

The set-up of this paper seems to have significant problems as both the search and the analysis have been performed using the same data-set (TEST – set) which actually a very small data set (years 2000-2005, only five years of monthly data for SP1 case).

The authors should have used a hold-out set to investigate the actual performance in “unseen” data.

In any case both the architecture selection, loss functions performance appear with really low F1 score values (below 0,3 in test set) – whereas the authors have stated earlier “we require that the accuracy on each class is at least 50%”.

The paragraph that presents the model architecture is not clear. How many layers and neurons do we have in the selected model(s)?

This performance cannot and should not be considered as appropriate for a forecasting model. Therefore, both models (Lisbon and Munich) cannot be used for drought prediction.

This is something that the authors actually acknowledge as they state “The precision of the prediction in both cases was rather moderate, as a high percentage of data is misclassified”.

The second half of the study presents the analysis of the performance obtained architectures.

In the 3.2.1 Shapely values section, we do not know which data set has been used – we assume that we are looking at the Test set. For Lisbon the cumulative contribution of the top 15 variables (out of the 27) is 20% which should explain the underperformance of the selected architecture. The case of Munich is even worse as the cumulative contribution of the top 15 variables is less than 5%.

Similar performance can be seen with seasonality analysis.

Last, in conclusion (line 290) the author state “Best performing models obtained accuracies of 57% for the Lisbon domain and 55% for the Munich domain”. This is not true as this performance has been seen in train set , not the test set. Even if it was in the test set it would have been insufficient as the model has already seen the information in the data set and therefore should not be considered for forecasting performance evaluation.

Citation: https://doi.org/10.5194/nhess-2021-110-RC1
- AC1: 'Reply on RC1', Elizaveta Felsche, 09 Jul 2021
  
  Authors would like to thank the anonimous reviewer for his/her useful suggestions and comments. Please referer to the supplement file for a point by point authors' reply to reviewer comments.
  
  Citation: https://doi.org/10.5194/nhess-2021-110-AC1
RC2:
'Comment on nhess-2021-110', Anonymous Referee #2, 17 Jun 2021

This study present a methodology for drought prediction at seasonal scale using machine learning algorithms. The study is treating a highly relevant subject with the usage of a novel methodology based on machine learning for droughts predictions. However, the issue with machine learning and climate is the length of the climate records, which does not allows to build AI models. Therefore, this study proposes a methodology fully based on a down scaled ESM. The study is interesting however, I have major comments listed bellow, in particular, at that stage it is very hard to evaluate properly the manuscript since the data/method is not clear enough:

1) the method (if I understood it correctly) is fully based on model data, therefor it is not really a study about prediction but according to me it is only potential predictability, since this study does not demonstrate any skill in predicting observed past climate in the two regions of interest, but only the ability to forecast the model climate. The paper should be much clearer about this, for example the title and the abstract should use the term “perfect model framework” and/or potential predictability.

2) The method description is very unclear about the prediction aspects. What are the target month analyzed? From which start date? For example, it is really confusing to me to predict SPI1 with one month lead time for different seasons. What do you mean here? Do you mix all together the start date of March (to predict the SPI1 of April), April (to predict the SPI1 of May) and May (to predict the SPI1 of June)? Or do you predict SPI1 integrated over MAM, but in this case, to my understanding it is not SPI1 but SPI3. In any case the methodology should be much clearer about this point, at this stage I cannot evaluate properly the manuscript without this clarification.

3) “The data from the years 1957 - 1999 was used as training data, the years 2000-2005 were used for the testing purpose.” Do you mean that the score calculation is performed only for 6 years from 2000 to 2005? This is a far too short period for any skill assessment. Usually, in seasonal prediction the skill is assessed over the whole hindcast period (1957-2005), using cross validation to construct the prediction.

4) The discussion does not mention at all the main limitation of this study according to me: at that stage the authors have demonstrated some ability to predict a model using AI, but we don’t know how to use such method for real prediction. Would it be possible to apply your model on observation and then verify its skill? If yes, it should be included in the study and if not this should be clearly mentioned.

Typos:

This study uses the monthly sea level pressure (pr)

The stong influence of ps/psl and NAO shows the influence of the atmospheric pressure

Citation: https://doi.org/10.5194/nhess-2021-110-RC2
- AC2: 'Reply on RC2', Elizaveta Felsche, 09 Jul 2021
  
  Authors would like to thank the anonimous reviewer for his/her useful suggestions and comments. Please referer to the supplement file for a point by point authors' reply to reviewer comments.
  
  Citation: https://doi.org/10.5194/nhess-2021-110-AC2

Peer review completion

AR: Author's response | RR: Referee report | ED: Editor decision

ED: Reconsider after major revisions (further review by editor and referees) (20 Jul 2021) by Athanasios Loukas

AR by Elizaveta Felsche on behalf of the Authors (16 Aug 2021) Author's response Author's tracked changes Manuscript

ED: Referee Nomination & Report Request started (17 Aug 2021) by Athanasios Loukas

RR by Anonymous Referee #3 (01 Sep 2021)

Suggestions for revision or reasons for rejection

The paper deals with an important issue in the field of changing climate and the use of machine learning methods to identify drought conditions. In my opinion, the paper has been prepared in a good manner and presents adequate technical and experimental details. The paper is a novel study and is generally well-structured as it explains the methodology, the mathematical framework and the assumptions used, and the justification of the results and the conclusions. However, the application research part needs improvements and corrections to verify the novelties of the method employed in the study area. Furthermore, there are few critical points that should be addressed in the revised manuscript. Addressing these comments will improve the quality of the paper and help the general reader of the paper. The paper could be accepted for publication considering the following revisions.

1. Why the authors use the parametric Pearson correlation (ρ) and not a non-parametric test (like Kendall tau and/or Spearman rho)? In case of nonlinear correlation between climatic signals and local drought values, statistics such as mutual information (MI) could be more informative than the conventional correlation coefficient. Therefore, I suggest to present mutual information as another statistic in Table 1. The authors state (lines 116-117) “The correlation coefficients reveal that out of the full list of 42 variables 14 are sorted out as being redundant”. Hence, nonlinear and/or non-parametric statistic values should be added to verify this conclusion.
2. Justification of the selected timescale of SPI and the selected forecasted lead-time. Why the authors use the SPI-1month? How reliable is the SPI-1 for drought prediction at the study sites? Why the previous 12 months are used as input for predicting the SPI of the next month (please provide scientific evidence for this assumption and why not testing up to 36 months before)?
3. The results clearly show (see Accuracy and F1-score) that the final selected models cannot be used for forecasting purposes for the selected drought index. The Threat Score (TS) or the Critical Success Index (CSI) could be used to verify if the proposed method should be used (https://www.cawcr.gov.au/projects/verification/Hewson/DeterministicLimit.html). Furthermore, I have the impression that the selected timescale of SPI (1-month) leads to unstable results. If the authors use for example the SPI-6 month how different would be the derived results?

Minor comments
1. A flow chart of the proposed method may also be added. The authors are requested to ensure that international readers/scientists will be able to apply this methodology on their data sets by following the flow chart.

For the motivations listed above, the paper in its present form needs revisions in order to evaluate the innovative character of the manuscript. However, the paper is of general interest for international audience and merits publication in NHESS Journal when the major and minor comments are addressed. Addressing these comments will improve the quality of the paper and help the general reader of the paper.

Hide

RR by Anonymous Referee #2 (17 Sep 2021)

ED: Publish subject to minor revisions (review by editor) (27 Sep 2021) by Athanasios Loukas

AR by Elizaveta Felsche on behalf of the Authors (25 Oct 2021) Author's response Author's tracked changes Manuscript

ED: Publish as is (26 Oct 2021) by Athanasios Loukas

Short summary

This study applies artificial neural networks to predict drought occurrence in Munich and Lisbon, with a lead time of 1 month. An analysis of the variables that have the highest impact on the prediction is performed. The study shows that the North Atlantic Oscillation index and air pressure 1 month before the event have the highest importance for the prediction. Moreover, it shows that seasonality strongly influences the goodness of prediction for the Lisbon domain.