Transferability of data-driven models to predict urban pluvial flood water depth in Berlin, Germany
- Chair of Hydrology, Institute of Environmental Science and Geography, University of Potsdam
- Chair of Hydrology, Institute of Environmental Science and Geography, University of Potsdam
Abstract. Data-driven models have been recently suggested to surrogate computationally expensive hydrodynamic models to map flood hazards. However, most studies focused on developing models for the same area or the same precipitation event. It is hence not obvious how transferable the models are in space. This study evaluates the performance of a convolutional neural network (CNN) based on the U-Net architecture and the random forest (RF) algorithm to predict flood water depth, the models' transferability in space and performance improvement using transfer learning techniques. We used three study areas in Berlin to train, validate and test the models. The results showed that (1) the RF models outperformed the CNN models for predictions within the training domain, presumable at the cost of overfitting; (2) the CNN models had significantly higher potential than the RF models to generalize beyond the training domain; and (3) the CNN models could better benefit from transfer learning technique to boost their performance outside training domains than RF models.
Omar Seleem et al.
Status: final response (author comments only)
-
RC1: 'Comment on nhess-2022-263', Anonymous Referee #1, 07 Nov 2022
This paper is concerned with the comparison of two machine learning techniques for predicting urban pluvial flood hazard in areas where the machine learning model was not trained. Namely, these are the UNET approach that has recently been used in a number of studies for the same purpose, and the random forest approach that was similarly used for predicting flooding, but (to my knowledge) not in a context of generating 2D flood maps for specific rain events. In addition, the papers explores the probability of transfer learning for these approachesÂ
The paper is interesting, generally well written and within the scope of the journal. I do have one major criticism and a number of minor comments as detailed below, but I expect that these can be adressed in a revision and that the paper can subsequently be recommended for publication.Major comments:
The comparison of UNET and random forest (RF) is a (the) key aspect of the paper. The paper concludes that the RF does not transfer well to other areas because it overfits the training data. This effect can and should have been avoided, and since it is such a central part of the paper I do think it needs to be fixed.
The straightforward approach would be to apply a cross validation approach. Divide the training dataset into, for example, 5 areas, perform 4 training iterations where you train on 4 areas and validate on the 5th, select RF hyperparameters that minimize the cross validation loss, and then train with these hyperparameters on the entire dataset.Â
In particular, this concerns the depth of the decision trees. As I understand from the paper, these have not been limited, and therefore the trees probably simply incorporate the entire training dataset. Please do include results for this in the paper or appendix.Minor comments:
- Figure 1: The different subareas have quite different properties (models trained on some areas generalize, while this is not the case for other areas). Please include som detailed illustrations of the subareas (elevation, maybe also flood areas). I think this Figure could become a 4 panel figure, one panel showing the overview of Berlin and the other panels details of the 3 areas. The legend needs to include units and an explanation that it is elevation that is displayed.
- line 92: what kind of storms were fed into the hydrodynamic simulations? block rains? CDS? Euler? ...
- line 173: please provide details on how feature importance was assessed (remove variables and assess relative change of loss function? validation loss or training loss?)
- Figure 4:
- This figure is impossible to read in black and white print
- While all the information in the figure is highly relevant, it is also quite convoluted and hard to understand. My guess is that it will be easier to read if you group all UNET results on the left, and all RF results on the right. "UNET" and "RF" could then be moved as headings to the top of the figure, making it a bit easier to recognize the "training domains" text as an axis caption (there is also a typo in the figure). You can then also use the same symbol for RF and UNET results, making the legend consistent with the symbols in the figure.
- Figure 5:
- I would suggest rearraning this figure in the same way as Fig. 4. Consider also including some more text into the figure, e.g. "baseline training areas" and "transfer test area" so that the figure becomes a bit more self-sufficient. Symbols like SA0->SA1&2 make it very hard to understand the figure without reading everything else in detail.
- Figure 6:
- Please include a similar figure for a smaller event. The most challenging part for machine learning models is to correctly capture the boundary between flood / no flood. In general, I miss results for how well the models perform for different rain intnsities.
Code (these are not review comments but it would be great if you could adress these):
The datafile on Zenodo is rather big. We tried to download it and failed - it would be great if you could subdivide the data into multiple zip files.Â
In addition, the modelling scripts start by loading pickle files, but the scripts generating these from the original data are not provided.- AC1: 'Reply on RC1', Omar Seleem, 19 Dec 2022
-
RC2: 'Comment on nhess-2022-263', Anonymous Referee #2, 16 Nov 2022
The comment was uploaded in the form of a supplement: https://nhess.copernicus.org/preprints/nhess-2022-263/nhess-2022-263-RC2-supplement.pdf
-
AC2: 'Reply on RC2', Omar Seleem, 19 Dec 2022
We would like to thank you for your comments and constructive suggestions for our manuscript. We very much appreciate the time and effort that you have invested in your reports. Please find our response in the attached supplement.Â
- AC3: 'Reply on AC2', Omar Seleem, 19 Dec 2022
-
AC2: 'Reply on RC2', Omar Seleem, 19 Dec 2022
Omar Seleem et al.
Omar Seleem et al.
Viewed
HTML | XML | Total | BibTeX | EndNote | |
---|---|---|---|---|---|
399 | 95 | 16 | 510 | 6 | 5 |
- HTML: 399
- PDF: 95
- XML: 16
- Total: 510
- BibTeX: 6
- EndNote: 5
Viewed (geographical distribution)
Country | # | Views | % |
---|
Total: | 0 |
HTML: | 0 |
PDF: | 0 |
XML: | 0 |
- 1