Multi-variable flood damage modelling with limited data using supervised learning approaches

Wagenaar, Dennis; de Jong, Jurjen; Bouwer, Laurens M.

doi:https://doi.org/10.5194/nhess-17-1683-2017

Articles | Volume 17, issue 9

https://doi.org/10.5194/nhess-17-1683-2017

© Author(s) 2017. This work is distributed under
the Creative Commons Attribution 3.0 License.

Special issue:

Damage of natural hazards: assessment and mitigation

https://doi.org/10.5194/nhess-17-1683-2017

© Author(s) 2017. This work is distributed under
the Creative Commons Attribution 3.0 License.

Articles | Volume 17, issue 9

Research article

|

29 Sep 2017

Research article |

| 29 Sep 2017

Multi-variable flood damage modelling with limited data using supervised learning approaches

Dennis Wagenaar, Jurjen de Jong, and Laurens M. Bouwer

Download

Final revised paper (published on 29 Sep 2017)
Supplement to the final revised paper
Preprint (discussion started on 12 Jan 2017)

Interactive discussion

Status: closed

AC: Author comment | RC: Referee comment | SC: Short comment | EC: Editor comment

- Printer-friendly version

- Supplement

RC1: 'Reviewer comment to HNESS-2017-7', Anonymous Referee #1, 10 Feb 2017
- AC1: 'Response to review Referee 1', Dennis Wagenaar, 21 Apr 2017
RC2: 'Report on nhess-2017-7', Anonymous Referee #2, 10 Feb 2017
- AC2: 'Response to review Referee 2', Dennis Wagenaar, 21 Apr 2017

Peer-review completion

AR: Author's response | RR: Referee report | ED: Editor decision

ED: Reconsider after major revisions (further review by Editor and Referees) (22 Apr 2017) by Thomas Thaler

AR by Dennis Wagenaar on behalf of the Authors (02 Jun 2017) Author's response Manuscript

ED: Referee Nomination & Report Request started (06 Jun 2017) by Thomas Thaler

RR by Sven Fuchs (08 Jun 2017)

RR by Anonymous Referee #1 (29 Jun 2017)

Suggestions for revision or reasons for rejection

The efforts the authors have put into the revision of the manuscript have improved the quality of the paper. However, I still feel that the paper would need further rework, especially as far as the methodological sections are concerned.

\section{Methodology}
Since this is a manuscript relying very much on a methodological comparison of different techniques, I feel that the methodology section is somewhat superficial and contains several aspects that remain unclear to the user.
Some examples:
\begin{itemize}
\item Have all variables been used as-is in the multiple linear regression model? I am wondering if transformations of certain variables might to better results in the multiple linear regression model?
\item Lasso does not only perform penalization/shrinkage, but also variable selection. Variable selection is not mentioned in this context, even though it probably is important in case of the multiple linear regression model as well. I find it somewhat difficult to believe that not a single variable is excluded by the Lasso, given that there are certain variables with correlation coefficients to the target variable of as low as 0.01 (Table 1).
\item Alpha values for Lasso are usually derived by means of cross validation. It is unclear why and how 4 specific alpha values were selected in this study. Lasso Trace Plots would be a nice graphical illustration that could be added to the manuscript. In addition, it is not clear how the "best" fitting lasso model was selected out of these 4 models.
\item Actually, scikit-learn does not support post-pruning, but pre-pruning is available (e.g. specifying depth of tree or split criteria, as implemented by the authors).
\item It is not clear to the user how pruning has been implemented. How does this work, apart from plugging the data into some Matlab toolbox? Has the tree size been selected according to the minimal CV error, or did the authors apply other methods to determine the tree size?
\end{itemize}

\section{General Comments}
\begin{itemize}
\item Conceptual confusion is created by somewhat randomly using metrics like (R)MSE, MBE and MAE. For instance, the authors state a 20\% reduction in the MAE in the abstract, while they state a 30\% reduction in the MSE in section 3.3.
\item I do not understand why has the RMSE been replaced with the MAE in section 3 and in the tables? The RMSE gives more weight to large errors than the MAE, thus being a nice metric if large errors are particularly undesirable. Why has this been dropped, instead of keeping the RMSE and simply adding the MAE (or rather, adding the RMSE, since the MAE has seemingly been only labelled incorrectly)?
\item Section 3.3: \em{The total importance of variables that were added in this study is about 30\%.} -- I assume this does refer to the test set? Please clarify.
\item
\end{itemize}

\section{Software References}
I would advise to refrain from justifying decisions based on the software used. While it is of course essential to reference the software used, providing explanations related to the methodology itself rather than based on some function arguments is way more proficient:
\begin{itemize}
\item \textit{This library requires an alpha parameter to be set (...)} -- actually, this is a core feature of Lasso rather than an attribute of the Python library.
\item \textit{This was done using Matlab's Statistics and Machine Learning Toolbox (...)} Ok, but how does this work? Which criteria have been used for pruning, i.e. how did the authors derive the optimally pruned tree? There is a reference to Breiman et al., but adding this essential information (1-2 sentences) to a manuscript targeted at a comparison of methods is important to the interested reader.
\item Section 2.3: It is sufficient to say that the variable importance has been calculated as the normalized total reduction of the MSE, there is no need to reference scikit-learn twice and simply base your decision on the fact that this is feature of scikit-learn.
\end{itemize}

\section{Figures}
\begin{itemize}
\item Figure 5: I would strongly advise to refrain from using pie charts, especially in the way they are presented here, since they are very difficult to read. There are too many variables displayed in random order, using very similar colors. Grouped bar charts or similar plot types would be preferable.
\item Figure 6: Please show the number of observations in the plot, e.g. above each box between y-values of 60000 and 70000. In addition, I am wondering why the construction of boxplots differs from the commonly used way of using the first and third quartile as box (plus a band inside the box indicating the median, which is also missing in this case). By the way, using the 5th and 95th percentile as whisker ends is not very common as well, but since it is indicated in the caption it is okay in this case.
\end{itemize}

Hide

ED: Reconsider after major revisions (further review by Editor and Referees) (29 Jun 2017) by Thomas Thaler

AR by Dennis Wagenaar on behalf of the Authors (25 Jul 2017) Author's response Manuscript

ED: Publish as is (30 Jul 2017) by Thomas Thaler

AR by Dennis Wagenaar on behalf of the Authors (17 Aug 2017)

Download

Article (1780 KB)
Full-text XML

Short summary

Flood damage models are an important component of cost–benefit analyses for flood protection measures. Currently flood damage models predict the flood damage often only based on water depth. Recently, some progress has been made in also including other variables for this prediction. Data-intensive approaches (machine learning) have been applied to do this. In practice the required data for this are rare. We apply these new approaches on a new type of dataset (combination of different sources).