the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
09 Feb 2021
09 Feb 2021
Leveraging multi-model season-ahead streamflow forecasts to trigger advanced flood preparedness
- 1Department of Civil and Environmental Engineering, University of Wisconsin-Madison, Madison, USA
- 2Nelson Institute for Environmental Studies, University of Wisconsin-Madison, Madison, USA
- 3Climate Hazards Center, Department of Geography, University of California, Santa Barbara, USA
- 4Red Cross Red Crescent Climate Centre, The Hague, 2521 CV, the Netherlands
- 5Universidad Tecnológica del Perú (UTP), Lima, Perú
- 1Department of Civil and Environmental Engineering, University of Wisconsin-Madison, Madison, USA
- 2Nelson Institute for Environmental Studies, University of Wisconsin-Madison, Madison, USA
- 3Climate Hazards Center, Department of Geography, University of California, Santa Barbara, USA
- 4Red Cross Red Crescent Climate Centre, The Hague, 2521 CV, the Netherlands
- 5Universidad Tecnológica del Perú (UTP), Lima, Perú
Abstract. Disaster planning has historically allocated minimal effort and finances toward advanced preparedness, however evidence supports reduced vulnerability to flood events, saving lives and money, through appropriate early actions. Among other requirements, effective early action systems necessitate the availability of high-quality forecasts to inform decision making. In this study, we evaluate the ability of statistical and physically based season-ahead prediction models to appropriately trigger flood early preparedness actions based on a 75 % or greater probability of surpassing the 80th percentile of historical seasonal streamflow for the flood-prone Marañón River and Piura River in Peru. The statistical prediction model, developed in this work, leverages the asymmetric relationship between seasonal streamflow and the ENSO phenomenon. Additionally, a multi-model (least squares combination) is also evaluated against current operational practices. The statistical and multi-model predictions demonstrate superior performance compared to the physically based model for the Marañón River by correctly triggering preparedness actions in all four historical occasions. For the Piura River, the statistical model proves superior to all other approaches, and even achieves an 86 % hit rate when the required threshold exceedance probability is reduced to 50 %, with only one false alarm. Continued efforts should focus on applying this season-ahead prediction framework to additional flood-prone locations where early actions may be warranted and current forecast capacity is limited.
Colin Keating et al.
Status: final response (author comments only)
-
RC1: 'Comment on nhess-2021-25', Anonymous Referee #1, 17 Feb 2021
General comments
The paper under review addresses an important topic within the scope of the journal, is generally well written and structured. Figures are visually appealing (especially Fig.6 & 7). Datasets used are adequate for the purpose of the study. Methods are rather traditional statistics (fairly old-fashioned), mainly a linear regression on principal components, but presumably also quite robust. No non-linear transformations, no unconventional predictors. The multi-model approach mentioned in the title is interesting. The chosen performance metrics for validation are also suitable. According to the authors, the developed model is an improvement to the current operational methods in Peru.
I suggest adding “in Peru” to the title of the manuscript – or any other spatial restriction the authors consider appropriate – as the method was only tested for two rivers in this specific country, and includes predictors that might not be suitable in other areas of the world (e.g. sea surface temperature for ENSO condition). If the authors want to claim that their method is in general better than operational practices worldwide, this claim would have to be substantiated by additional model runs in different places.
The authors made their code available to review via a GitLab repository, which is much appreciated! The provided R scripts are well readable (although not entirely in agreement with modern style guides, e.g. https://style.tidyverse.org/) and seem to cover all steps mentioned in the manuscript, from data preparation to model building and plotting. I did not try to run the code, as the raw data is not provided, but the scripts make the conducted research transparent.
Specific comments
About the manuscript, I request the following clarifications and modifications:
- Please clearly define the term “season-ahead prediction”. The term could be interpreted as predicting one season from the previous season, but I assume that the authors mean to predict one season from just before the start of that very season, as the 1-month-ahead streamflow appears to be included as predictor. Does the model only predict the maximum streamflow at some point during the season, or also a timing? 3 months is still quite an uncertain timeframe.
- In the introduction and discussion there should be an additional paragraph putting the used methods in context of what is state of the art in international scientific literature – not only in Peru. The last two sentences of the conclusion are: “(…) because the statistical model developed here is optimized for performance across all years, further refinement prioritizing the detection of appropriate trigger levels for early action in high flow years may be warranted. Such efforts could involve alternative modeling frameworks (e.g. logistic regression), additional predictors, and evaluation of category selection applied in the prediction process.” - But that is not enough and should appear earlier in the paper. Also, an additional paragraph about ensemble theory / multi-model studies would be adequate.
- Data: The authors should make very clear for the reader which data was used to fit the statistical models, i.e. how many observations, where does the target variable (y) come from and how certain is it, what exactly are the explanatory variables and how have they been treated (scaling etc.). Most of that information is somewhere in the manuscript, but it is not as clear as it should be on first reading. Table 3 could be a good place to collect this information.
- “There are numerous methods for selecting the appropriate number of PCs to retain; here, the first two PCs are retained unless the model has two or fewer predictors, and then only the first PC is retained.” (254-256). How is the selection of only 2 PCs motivated? Contributions may differ during the seasons or per region, but at least some sort of check should be presented, e.g. by plotting the cumulative explained variance for El Niño and La Niña (or any other method the authors prefer to make this point that 2 PCs are sufficient). According to Table 3, only in one case have there been 2 PCs used – in all other cases only 1, so it is only linear regression with 1 predictor? Or 1 PC plus the streamflow before the start of the season?
- A critical point, acknowledged by the authors, is the selection of a threshold to issue an emergency. In my opinion this problem could be communicated better to the decision makers if a full probability distribution of expected streamflow were predicted, rather than a point estimate. Bayesian regression would be the adequate tool, then. As the statistical model presented by the authors appears to be very simple (linear regression with 1 or 2 predictors), implementing this in a Bayesian framework should be feasible. In that case, also Bayesian decision theory could be applied for the threshold selection. Apparently the authors create an error distribution by sampling the model residuals 1000x with replacement, which might end up in similar estimates, although with slightly different interpretation. At least the authors should discuss the probabilistic output in more detail, and also discuss how this probabilistic output can be used in risk communication and decision making for the problem at hand.
- The multi-model seems to be dominated by the linear regression model. If this is the case, the authors could discuss which other models might be suitable to include in future multi-model ensembles.
Technical corrections
- All tables would benefit from some formatting.
- In Table 2, the letters J and F are used without explanation. I assume it is January and February, respectively, as the authors write in the text that the high streamflow seasons in the basins are FMA and MAM, respectively. January and February would therefore correspond to a 1-month-ahead value. However, that should be stated explicitly in the text and above the table – or more clear abbreviations like “Jan” and “Feb” should be used.
- Especially the very important “predictors” column in Table 3 consists of abbreviations with distracting line breaks. As the columns of that table are repeated, consider arranging the “negative phase” “positive phase” and “neutral phase” in rows rather than columns, and use the free space to add more columns giving detailed information on the models, like the number of observations, PC2 explained variance, maybe even the cross-validation score. Consider removing the bold rectangle and make the font of the column/row names bold instead.
- There is a LICENSE file included in the GitLab repository, but no README and CITATION files. I would like to encourage the authors to add these two missing components, although it is not a criterion for acceptance of the manuscript.
- AC1: 'Reply on RC1', Colin Keating, 29 Mar 2021
-
RC2: 'Comment on nhess-2021-25', Anonymous Referee #2, 01 Apr 2021
This paper describes evaluation of seasonal flood forecasts over Peru. The results are a key part of developing forecast-based early warning systems, where a robust understanding of model skill is crucial. The paper addresses an important question, the results are interesting and the manuscript is well structured and clear.
I am happy to recommend publication, after the authors address a few comments, below.
1. False alarm ratio (FAR) is calculated by counting #false alarms and #triggers and dividing the former by the latter. This is fine and is the standard method to do so. However it does mean that the sample is relatively low (particularly for seasonal reforecasts), leading to high uncertainty on the values. e.g. The sample size at Maranon is 19, but there are many fewer flood events and triggers than this. It is possible to partially address with uncertainty ranges on the verification statistics, calculated through a standard bootstrap resampling method (i.e. pick a 19 years with replacement from Maranon, recalculate FAR/ HR/ POD, and repeat). I would like to see this uncertainty added to figure 8, and its implications discussed.
2. In addition to point 1 above. The sample size means there is an inevitable an aspect of forecast behaviour which is not captured in the reforecast - and bootstrap resampling across years is unable to quantify this. To explain with an example: in 2013 the statistical model predicts 94% probability of exceedence, which is observed. In the evaluation this year will always turn up as a 'hit' for any threshold less than 94%. However if we take the probability as a reliable representation of likelihood, there is still a chance that it would have been a false alarm (i.e. 6%). Similarly, there is a chance that every probability which resulted in a trigger was a false alarm (as long as that probability wasn't 100%). This is an unavoidable result of small sample size - and one which bootstrapping will not quantify - so I am not suggesting any change. However I suggest the authors reconsider their conclusion "L483 Detection of additional high flow events is possible by lowering the forecast probability ... while maintaining a low false alarm ratio". This is only true for this particular realisation of the reforecast. If you lower the probability threshold, there will always be more chance of false alarms when you trigger, by definition. You might get lucky, but then again you might not. It is important to be clear about this otherwise misleading conclusions may be reached, e.g. L427 suggests a lowering of the trigger to 50% may capture many more events, "without additional false positives". This is highly contingent on the particular realisation of the reforecast. A decision-maker may read this paper and decide to take action when the forecast probability is 50%, as they understand that this has an FAR of 0%. But, the chance that action on a forecast of exactly 50% will be in vain is ... 50% (assuming the probability is reliable). So there is a good chance they may be in for a nasty shock! I suggests the authors rethink their advice on lowering the trigger without consequence.
3. The statistical model uses antecedent SST as a predictor (capturing ENSO activity). It also uses a precipitation forecast from the NMME. But what about using the SST forecast from the NMME? If ENSO state is a strong forcing of rainfall/streamflow, then I would imagine that the FMA SST is more strongly related to streamflow than DJF SST? Possibly the precipitation forecast may capture some of this future signal - although precipitation errors are well known. I hope that the authors might consider adding this, as it may increase the skill even further and lead to a better early warning.
4. Can you show the weightings for the statistical model? The results are shown from cross-validation leave-one-out (which is appropriate). But if you built the model again using all years, this would be useful to show the relative importance of each predictor.
5. The GloFAS seasonal forecasts are publicly available on the 10th of every month - not the first, as is stated in L274 (see https://www.globalfloods.eu/technical-information/glofas-seasonal/ - NB they are initialised on the 1st but there is a lag until they are available, which may be where the confusion arises). Does this change the potential for early action, as the first month is almost half over before the GloFAS forecast is available? There are a few possibilities:
- if the action is strictly constrained to the start of the month, the GloFAS run from the previous month is the only available run, so this should be used instead in the comparison
- if it is OK that no forecast is available until the 10th, then the statistical model could (in theory) include additional information on the streamflow/SST/precip in the first few days.
The authors may want to follow either (or neither) of these ideas. But at least please comment on this issue of forecast timeliness in the text.Minor comments
L41 Was FbA originally applied to droughts? As far as I am aware it is only now being developed for drought/food insecurity. Please clarify.
L69 There is a bit of a logical jump from the previous paragraph, consider adding a linking sentence.
L111 Slightly long sentence, could be split for readability.
L160 What do the colours represent in Figure 1? Satellite image, topography? If the latter then it needs a colorbar.
L200 Table 2: Piura has correlation of 0.84 between J streamflow and FMA streamflow. However in L148 it states that there is no significant monthly autocorrelation in Piura streamflow. This seems to be inconsistent.
L200 Table 2: Maranon GCM precipitation forecast is not included as a predictor, presumably because the correlation with MAM streamflow is not sufficiently high. I wonder: is this because (a) there is low correlation between seasonal rainfall and seasonal streamflow at Maranon or (b) the GCM precipitation forecast at Maranon is not particularly good? It would be good to include this information. If the answer is (b), then see point 3 above: it may be that SST is a more valuable predictor to take from the GCM forecast.
L226 I am unsure what " n.d." means in this context.
L228 A 3-phase ENSO model is used at Piura, although a 2-phase model does not affect material performance. Given the favouring of parsimonious models (L257), why do you retain the 3-phase model?
L272 Requires some more info on GloFAS: what is the reforecast period, which model version used, has the model been calibrated for these basins (where streamflow data has been shared with the GloFAS team, the model has been calibrated).
L315 It would be useful to explicitly note how many upper tercile events are present for each site.
L399 What is meant by 'observed trigger'? From the context I think it should read 'event'? 'Trigger' only applies in context of the forecast, not the observations (similarly used in L448).
L450 I am not sure what is meant by "TS is maximised".
L463 Another thing to consider with these close-to-threshold events is that the difference in streamflow between may very well be within the margin of observational error - particularly if the seasonal average is based on daily data (i.e. an accumulation of systematic/random errors over 90 days).
L468 "two events of similar magnitude...are likely to produce similar impacts with early actions likely to yield similar benefits". I am not sure it is reasonable to say this. Two seasons with similar average seasonal streamflow may have highly different subseasonal variability. For instance season A: all season just below the overtopping level without breaching, season B: a little way below season A average for the first month, but then increasing and repeatedly flooding in the next two months. A & B may have very similar average streamflow - but very different impacts.- AC2: 'Reply on RC2', Colin Keating, 14 May 2021
Colin Keating et al.
Model code and software
Peru Streamflow Prediction Colin Keating https://gitlab.com/ckeating/peru_streamflow_prediction
Colin Keating et al.
Viewed
| HTML | XML | Total | BibTeX | EndNote | |
|---|---|---|---|---|---|
| 261 | 67 | 10 | 338 | 1 | 2 |
- HTML: 261
- PDF: 67
- XML: 10
- Total: 338
- BibTeX: 1
- EndNote: 2
Viewed (geographical distribution)
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1