the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
| 16 Sep 2024
Evaluating Yangtze River Delta Urban Agglomeration flood risk using hybrid method of AutoML and AHP
Abstract. With rapid urbanization, the scientific assessment of disaster risk caused by flooding events has become an essential task for disaster prevention and mitigation. However, adaptively selecting optimal machine learning (ML) models for flood risk assessment and further conducting spatial and temporal analyses of flood risk characteristics in urban agglomerations remains challenging. This study, establishes a "H–E–V–R" risk assessment index system that integrates hazard, exposure, vulnerability, and resilience based on the factors influencing flood risk in the Yangtze River Delta Urban Agglomeration (YRDUA). Utilizing Automated Machine Learning (AutoML) and the Analytic Hierarchy Process (AHP), a comprehensive flood risk assessment model is constructed. Results indicate that, among those of different assessment models, the accuracy, precision, F1-score, and kappa coefficient of the CatBoost model for flooded point identification are the highest. Among the flood hazard factors, elevation ranks highest in importance, with a contribution rate of up to 68.55 %. The spatial distribution of flood risk in the study area from 1990 to 2020 is heterogeneous, with an overall increasing risk trend. This study is of great significance for advancing disaster prevention, mitigation, and sustainable development in the YRDUA.
- Preprint
(1453 KB) - Metadata XML
- BibTeX
- EndNote
Status: open (until 03 Dec 2024)
-
RC1: 'Comment on nhess-2024-144', Anonymous Referee #1, 13 Oct 2024
reply
The authors have presented an AutoML framework for calculating flood risk. AutoML is a compelling approach because it requires little effort to implement and ML models are automatically trained, often with fairly good performance.
However, using AutoML does not forego rigorous analysis, which I believe is lacking in this manuscript. I have expressed my concerns below:1. The datasets are not sufficiently described to be able to evaluate the manuscript’s results. Specific comments are provided below for improving dataset descriptions.
2. The results are not presented with sufficient level of detail to evaluate their validity. Specific comments are provided below.
3. It is not clear why the authors have used both AutoML and AHP, and how (or if) they compliment each other.
4. Line 57 - Sentence could not be understood.
5. Line 76 - Additionally, ML *can* not be fully automated…
6. Line 178 - It is not apparent why uneducated people would be more/less vulnerable to floods. Could the authors include why they chose this factor, instead of, let’s say, the average income distribution?
7. Line 179 - Please define urbanization rate.
8. Line 187 - Which flooding events were used for the labels?
9. Line 188 - How were the features, which are available at different resolutions, mapped to the meteorological stations to construct the training dataset?
10. Line 194 - It is not clear why data balancing is needed for small datasets. It is generally needed for imbalanced datasets.
11. Line 201 - Are the 278 points per label representative of the entire dataset or just the training dataset? It is recommended to not under/oversample the test data, but only the training and/or validation datasets.
12. Line 201 - What is the original distribution of the dataset labels?
13. Line 231 - The definitions of FN and FP are reversed. An FP would be a non-flooded point predicted as flooded by the model, and vice-versa.
14. Line 290 - It will be helpful to define consistency.
15. Eq 11 - lambda is undefined, as only lambda_max was defined in eq.10
16. Line 299 - Sentence in unclear.
17. Line 316 - Please clarify whether the metrics have been provided for the training data or the test data. Since the test data is expected to be imbalanced, accuracy would not be a good metric.
18. Line 319 - What is the probability threshold used for calculating precision and recall?
19. Line 327 - How was overfitting evaluated?
20. Line 368 - I would suggest citing Lundberg and Lee, 2017 instead of Wang et al., 2023a for the description of SHAP.
21. Figure 8 - What is the source of the figures?
22. Line 450 - What is causing the differences in flood risk over the decades.Citation: https://doi.org/10.5194/nhess-2024-144-RC1 -
AC1: 'Reply on RC1', Shuliang Zhang, 27 Oct 2024
reply
Dear Editor and Esteemed Reviewers,
I would like to sincerely thank you for your thorough review and the valuable feedback provided on my submitted manuscript. I truly appreciate the time and effort you have dedicated to evaluating the paper. Every suggestion has been carefully considered, and I have made revisions accordingly, which are clearly marked in the revised manuscript. Below, I provide a detailed response to each of the reviewer’s comments.
We feel we have comprehensively addressed the concerns raised, and will leave how to progress our paper to your judgement. I am looking forward to hearing your positive response soon.
Sincerely.
1. The combination of machine learning and AHP methods for flood risk assessment is already quite common. However, using only a single machine learning algorithm tends to result in poor interpretability of flood risk, leading to uncertainty in the model’s flood risk results. Additionally, further research is needed to efficiently and accurately select the optimal machine learning algorithm for the region.
Automatic machine learning tries to automatize the steps of feature extraction, model and algorithm selection, parameter optimization, and so on so that it needs no human assistance and avoids man-made bias. This approach only requires the configuration of different run times, allowing the algorithm to explore a wider array of model and parameter combinations within the allocated time, ultimately leading to the identification of the best-performing model.
This paper selects the Auto-Sklearn framework to address the binary classification problem of flooded point identification and to calculate flood risk. By utilizing the characteristics of automated machine learning, the efficiency of machine learning is improved, and the importance ranking of flood hazard factors is obtained. The next step is to use the AHP to calculate the relevant weights by combining the flood risk results with exposure, vulnerability, and resilience indicators. AHP aims to quantify the decision-making process by scoring weights according to their level of importance, ultimately yielding the flood risk results.
2. Line 57 - Replace the sentence in line 57 with: " Through continuous improvement and development of machine learning algorithms,ensemble methods have been addressed the limitations of traditional machine learning models. "
3. Line 76 - The meaning expressed in line 76 is inaccurate; a more precise phrasing would be: “The effectiveness of machine learning "automatically improves with experience," and a key challenge in the research is how to integrate the data processing capabilities and feature selection strengths of hybrid models with ensemble models.”
4. Line 178 - The reason for choosing uneducated individuals as one of the indicators is that they are often part of socially vulnerable groups. People with lower levels of education may not fully understand warning information, disaster prevention measures, or have access to sufficient disaster preparedness resources, which increases their vulnerability during disasters. Additionally, those with higher education levels typically have access to more information channels, while individuals with lower education levels may be unfamiliar with new technologies or information channels (such as mobile apps and internet alerts). At the same time, lower-educated groups may live in areas with less developed infrastructure, making it difficult for them to receive timely social aid and support. Uneducated individuals may also find it harder to regain economic independence after a disaster, as they may lack access to technical training or knowledge updates, leading to slower recovery. Although income level is an important factor in assessing vulnerability, average income distribution can sometimes obscure individual differences. For example, a region may have a high average income level, but low-income groups (such as uneducated individuals) can still be in a highly vulnerable state. Additionally, income level may not directly reflect an individual's awareness of disaster, knowledge reserves, or ability to take action.
5. Line 179- Urbanization rate refers to the proportion of the urban population to the total permanent population in a given region, and it reflects the level of urbanization in that area. This indicator has an inverse relationship with flood vulnerability. Generally speaking, the higher the urbanization rate of a region, the higher the level of social development and the capacity for protection, which can reduce flood vulnerability to some extent.
6. Line 187- The flood inventory map in this paper was developed using inundation data from the Global Flood Database and flood disaster data from the EM-DAT database, supplemented by satellite and Google image interpretation and verified against existing historical flood records. The actual flood-affected areas were delineated based on flood traces from the inundation datasets and image interpretations. For this study, 278 flood inundation points were randomly selected within the inundation data range during the study period, and the location of each point was used as the basis for subsequent statistical analysis of flood events.
7. Line 188- For the collected data related to flood disaster risk, differences in sources and formats have resulted in variations in spatial resolution, dimensions, and magnitudes across the datasets. When using these data as indicators to assess risk, it is necessary to standardize the spatial and statistical data. This involves two key aspects: unification of spatial scale and normalization of numerical range.
(1) Unification of spatial scale means aligning data within the same coordinate range and resolution. The research data is standardized through projection transformation, converting all datasets into the same geographic and projected coordinate systems. The Kriging interpolation method is used to spatially process all discrete data. Finally, if the spatial data has different resolutions, resampling is performed to standardize all data to the same resolution, which in this study is unified to 30m×30m.
(2) Normalization of the numerical range can be achieved using a normalization process. Through a linear transformation, the values of the data are mapped to the range [0, 1], thus eliminating the influence of differing dimensions among the data indicators. In this study, the Min-Max Normalization method is used for normalization, and the formula is as follows:
x^'=(x-min(x))/(max(x)-min(x))
8. Line 194 - Through experiments, it was found that the effect of using a 1:1 ratio for the training and testing datasets is better than the 1:2 and 1:3 ratios.
9. Line 201- Each label consists of 278 points representing entire dataset.
10. Line 201- The overview map of the study area has been revised to display the spatial distribution of flooded and non-flooded points.
11. Line 231- The text here contains a definition error, which has been corrected in the main body of the paper. Thank you to the reviewer for pointing this out.
12. Line 290- The definition of consistency has been added to the paper: In a pairwise comparison matrix, the decision-maker's judgments must exhibit logical coherence and transitivity. This means that if option A is considered more important than option B, and option B is considered more important than option C, consistency requires that option A must also be judged more important than option C.
13. The λ in Eq. 11 should be λmax as defined in Eq. 10, and this has been corrected in the paper.
14. Line 299 - This sentence has been revised: Where average Random Consistency Index (RI) represents the average random consistency which depends only on the order of the judgment matrix. The RI values for judgment matrices of order 1 to 10 are shown in Table 3.
15. Line 316 - These indicators are provided for the testing data. In response to the reviewer’s suggestion, we will consider removing the accuracy metric and retaining precision, recall, F1-score, and Kappa.
16. Line 319 - The probability thresholds for accuracy, precision, recall, and F1-score range from [0, 1], while the Kappa coefficient ranges from [-1, 1].
17. Line 327 - We used 5-fold cross-validation to assess overfitting by comparing the performance of the training and testing sets. The experimental results indicate that the performance of the training set and testing set is relatively close, suggesting that the model does not exhibit overfitting.
18. Line 368 - This sentence has been revised: SHAP is an explanation method based on game theory and belongs to post-hoc model interpretation methods. SHAP values provide a unified framework that decomposes the model’s output to quantify each feature's contribution to the prediction result.
19. Figure 8 - Using AutoML and AHP, the flood hazard, exposure, vulnerability, and resilience of the YRDUA were calculated. Based on model-determined weights, flood hazard level (a), flood exposure level (b), flood vulnerability level (c), flood resilience level (d), and flood risk spatial distribution (e) were derived through natural breaks classification in ArcGIS software, resulting in a flood risk zoning map for the Yangtze River Delta region.
20. Line 450 - The differences in flood risk among cities in the Yangtze River Delta over the past few decades are primarily due to a complex interplay of various factors, including geographic and climatic conditions, urbanization processes, socio-economic factors, ecological changes, and historical flood events. The topography and precipitation patterns of different cities affect their capacity for rainwater drainage and accumulation, while urbanization leads to an increase in impervious surfaces and variations in infrastructure development, impacting flood management capabilities. Additionally, differences in population density, economic development levels, and flood management policies can exacerbate flood risk. Furthermore, the increasing frequency of extreme weather events due to climate change further elevates flood risk. These factors determine the varying levels of flood risk among cities within the Yangtze River Delta.
-
AC1: 'Reply on RC1', Shuliang Zhang, 27 Oct 2024
reply
Viewed
| HTML | XML | Total | BibTeX | EndNote | |
|---|---|---|---|---|---|
| 109 | 40 | 65 | 214 | 6 | 6 |
- HTML: 109
- PDF: 40
- XML: 65
- Total: 214
- BibTeX: 6
- EndNote: 6
Viewed (geographical distribution)
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1