a) Dataset. 

For better prediction results, the authors should preserve (and demonstrate) the original data distribution from the initial dataset when merging and filtering instances.

For the same reasons, this reviewer believes that it is necessary to include the class of undamaged buildings in the dataset (with 0$ compensations), which is unavoidable when mapping damage states.

Thanks for the comment.

This model has been developed for insurance purposes with the aim of helping the Earthquake Commission (EQC) to get an understanding of the possible loss distribution across Christchurch for any future earthquake. EQC’s interest is concentrated on damaged buildings for which a claim might be lodged. It must be reminded that the data used to train the machine learning (ML) model pertains to buildings for which one or multiple claims have been lodged as part of the Canterbury Earthquake Sequence (CES). Getting details information on undamaged buildings was not part of the scope. Thus, extracting reliable data related to the undamaged category is not straightforward. Simply assuming that a building was not damaged because no EQC claims were lodged is not satisfactory, as it is not possible to affirm that the remaining buildings had proper insurance coverage. Moreover, for some buildings which suffered slight damage, the building owners might have decided to cover the cost of the reparations by themselves to avoid paying the excess.

Nevertheless, following the suggestion, we tried to include a fourth category for undamaged buildings and explored the influence on the ML model performance. The EQC data includes a few instances with zero compensation (BuildingPaid = NZ$0). Figure i shows the number of instances for 4 Sep 2010 and 22 Feb 2011. With 7%, the number of instances is very limited compared to the “low” and “medium” categories. Despite the low number in the category with no damage, a new machine learning model has been retrained (considering class imbalance). Figure ii shows the confusion matrix for the Random Forest algorithm trained using four categories. It can be seen that the overall accuracy dropped and that the model is having difficulties making predictions for the zero damage category. It was thus decided to keep only three categories (low, medium, and overcap).

Figure i: Number of instances in BuildingPaid categorical in the filtered data set including the category with zero compensation: (a) 4 Sep 2010, (b) 22 Feb 2011

Figure ii: Confusion matrix for Random Forest algorithm including the category with BuildingPaid = 0

b) Building representation. 

For a better presentation of the mapping problem, the authors should show the data distribution for more features (like in Figure 6 for construction type).

Surprisingly, the height of buildings isn’t included in the building representation even though it captures the dynamics of the building. A feature like “number of floors” could possibly be informative.

Re: data distribution

Thanks for the suggestion. A new table has been added showing the nine attributes used in the model. The table gives information about the type and distribution of each attribute.

Re: height of the buildings

Thanks for the remark. The authors of this paper are aware of the inclusion of the building height (or number of stories) as an attribute in the ML models in similar studies (Ghimire et al., 2022; Harirchian et al., 2021; Mangalathu et al., 2020; Stojadinović et al., 2022). The non-inclusion of the building height in this study was dictated by the availability of the information in the dataset and not based on a deliberate choice. While the original EQC data has an attribute for the number of storeys, many instances are missing. Figure iii shows the number of instances available for each storey category. It can be seen that for 4 Sep 2010 and for 22 Feb 2011, storey information is missing for 58% and 93% of the instances respectively. The information available related to the building height is thus very limited. When available, the data for 4 Sep 2010 shows that 10,37 buildings have one storey, 2,677 two storeys, and 140 three storeys. Similarly, for the 22 Feb 2011, 1,522 of the buildings have one storey, 727 two storeys, and 91 three storeys. Selecting only instances where the number of storeys is known would have been very limiting from the aspect of training a ML model. It should be reminded that the EQC cover is limited to residential dwellings. This study is thus limited to residential buildings only which for Christchurch are mostly one-story height houses.

Accurate information on the number of storeys could not be obtained from the RiskScape database either. The attribute ‘Storeys’ is reported as a float which seems to have been calculated from the building floor area 'BLDGFL_1' divided by the building footprint 'FOOTP_1'.

In the aim of retaining high data accuracy it has been decided to not include the number of storeys from the EQC dataset in this study. This has been clarified in section 3.2 of the paper.

Figure iii: Number of instances for each storey category (EQC data): (a) 4 Sep 2010, (b) 22 Feb 2011

Re: model implementation for new earthquakes

Thanks for the comment. The ML model presented in this paper was designed to be easily retrainable. This enables the addition of new instances in the training set after the occurrence of a new earthquake. However, one of the challenges, is that claim amounts are not readily available after an earthquake as on-site assessment of building damage can be spread over a long period of time. To circumvent this issue, building damage should be assessed on a representative set of buildings. This subset where the damage extent will be known should be added to the training set of the ML model that can be retrained. The ML model can then be used to make predictions on the entire building portfolio. This approach has been schematically described in a new version of Fig 10.

The selection of a representative training set can be made following an event based on the effects of the earthquake or prior to the event using a predetermined representative subset of the residential building in Christchurch. Special care should be applied to ensure that the selected buildings can be used to produce a satisfactory seismic loss assessment at the city level. Recent discussions with experts highlighted the uniqueness of the Canterbury region. They mentioned that the analysis of damage observations across the CES showed that the main earthquake events affect different areas in Christchurch. They thus suggested that the selection of a representative set of buildings should take into account the geographical characteristics, the liquefaction setting, and building characteristics. 

When expert opinion is not available, similar studies showed that ML could even be employed in the selection of a representative building set (Mangalathu & Jeon, 2020). 

The actual selection of a representative set of buildings for Christchurch is beyond the scope of this study.

Re: Change in value of money

Thanks for the remark. The authors agree with the necessity to consider the evolution of the market over time. Here again, the ease of retraining the ML model when new or updated training data gets available should be highlighted. The authors are aware of the step change related to the value of the EQC cap over time (e.g., since 1 Oct 2022, the new cap is at NZ$300,000 + GST (Earthquake Commission (EQC)). Nevertheless, Fig 8 showed that for 4 Sep 2010 earthquake most of the claims fell in the ‘low’ and ‘medium’ categories. Even for the 22 Feb 2011 earthquake, which was unprecedented in the damage extent caused in the Canterbury region, many claims still relate to the ‘low’ and ‘medium’ categories. It is thus believed that the value of the model lies in its ability to make predictions for those categories (‘low’ reflecting the limit of initial cash settlement consideration, ‘medium’ for building having more damage but where claims are still fully addressed by EQC only, and ‘overcap’ where private insurer come into consideration for higher level of damages).

Re: no undamaged buildings class

Thanks again for the comment. Please see our reply to comment #1. This ML model has been developed with the purpose of being used in the insurance setting. The focus is thus on being able to predict the possible damage and loss extent for buildings having EQC claims in future earthquakes. 

Thanks for having highlighted those typos. The errors have been corrected.

Thanks for the comment.

The selection of the training, validation, and test set has been revised. The updated version is schematically shown in Fig 10. The training and the test set are now sourced from the same earthquake. The validation set is implemented using k-fold cross-validation as part of the hyperparameter tuning. Four models have been trained and tested using data from the four main earthquakes in the CES (4 Sep 2010, 22 Feb 2011, 13 Jun 2011, 23 Dec 2011). 

Figure i: New Figure 10

Thank you for the suggestion.

As mentioned in the reply to comment #1, the process of selecting the training, validation, and test set has been reviewed. The training and test set are now clearly separated thus ensuring that no data leakage can occur. Additionally, the way the model should be applied to future earthquakes has been clarified in Fig 10 as well as in the paper.

We are here deliberately focusing on the claim data related to the four main earthquakes rather than the claim data pertaining to the aftershocks. Figure ii shows the situation in the area around Christchurch as of December 2012 (one year after the end of the CES). It can clearly be seen that the earthquakes with the larger magnitudes were: 4 Sep 2010, 22 Feb 2011, 13 Jun 2011, 23 Dec 2011. Severe buildings damage, and hence higher claims, are mainly related to those four earthquakes as the seismic intensity and liquefaction occurrence were larger for those events, hence the selection of those earthquakes in this study.

Figure ii: Map of the region around Christchurch showing the epicentre location of the four main earthquake and multiple aftershocks in the Canterbury earthquake sequence (CES) (O’Rourke et al., 2014, originally from GNS)

Thanks for the comment.

As studies applying machine learning in an earthquake engineering context (see section 2.1 of the paper) are now more common, many publications include background explanation on theoretical part related to ML (e.g., Harirchian et al. (2021)). The authors are aware of the different metrics related to a classification (Figure iii) but decided to not include generic explanations related to ML to keep the paper to reasonable length.

One of the main reason for conveying the model performance using the accuracy was to enable the benchmarking of this ML model against the performance of other ML models for damage prediction. While a direct comparison would be improper as the earthquake selected, model attributes, and algorithm are not the same, most of the current studies report the model performance using the accuracy (Ghimire et al., 2022; Harirchian et al., 2021; Mangalathu et al., 2020; Stojadinović et al., 2022). 

Nevertheless, following the suggestion, the ML model is currently being retrained with emphasis on ‘recall’. We will observe the outputs and possible benefits and update section 8 of the paper accordingly.

Figure iii: Details of a confusion matrix for a binary class problem

Thanks for the remark.

Thanks for the note. The typo has been corrected.

Thanks for the suggestion.

Unfortunately, the EQC data is not public. We signed a confidentiality agreement with EQC which made the data available to us for research purposes only.

Thanks for the comment.

In some cases, multiple claims might have been lodged for the same earthquake event. The clarification has been added in the paper “For a particular earthquake event, it sometimes happens that multiple claims pertaining to the same building were lodged. Figure 2 thus specifically differentiates between the number of claims and the distinct number of buildings affected. “

The building characteristics data was obtained from the RiskScape - Asset Module Metadata (RiskScape, 2015). A copy of the document has been attached in Appendix. According to the documentation the “data in this inventory is partly derived from information purchased from Quotable Value (QV) Ltd, together with ‘industry knowledge’ and data gathered from surveying the area. All QV data is applied at the meshblock level, and the RiskScape attributes are derived from this information so as to provide a suitable model of the actual building stock.”

Further information can also be found in (King & Bell, 2006; Reese et al., 2007).

Thanks for the comment.

The paragraph has been rewritten to include explanations related to the location of the EQC and RiskScape coordinates. It is now clearly stated in the paper that the coordinates provided as part of the EQC dataset relate to the location of the street address while the RiskScape coordinates are located in the actual centre of the footprint of a building. In some cases, buildings from neighbouring properties are located closer to the street address than the actual building. For those cases, the use of spatial join functions and spatial nearest neighbour joins led to unsatisfactory outputs.

Thanks for the comment.

As suggested in comment #18, the percentages have been added to Table 1.

Thanks for the comment.

A comment mentioning that 27.1% of the RiskScape instances were having 3 or more RiskScape datapoints within a LINZ property title as been added at the end of the phrase. Additionally, as suggested in comment #18, the percentages have also been added to Table 1.

Thanks for your comment.

Fig 3 showed that some LINZ property titles included more than one point (i.e. building) per property title (polygon). This poses challenges for an automated data merging process as it needs to be ensured that the RiskScape and EQC information get assigned to the correct building.

As mentioned in line 121: “The merging process was thus started with instances having a unique street address per property”. Those instances having a unique building per polygon were used to merge RiskScape attributes and EQC info to the buildings constraining the merging within a property title. The actions for merging scenarios related to property titles having 1 point per LINZ property are listed in Table 1 rows #1 to #3.

It is known that the training of an ML model is often benefiting from more training data. Thus, options to merge more data were explored. It was found that among all the LINZ data, 7% of the LINZ property titles have two street address points (e.g. two buildings within one polygon). We explored if there would be an automated way to merge RiskScape attributes for such cases and thus get more data. For instances having 2 buildings (LINZ points) per property and 2 RiskScape points, the automated merging was done joining the RiskScape attribute to the closest LINZ point and filtered using the building floor area. Among the instances where a LINZ property title had 2 buildings it was also found that some of the RiskScape data only had one point per property title. While a human could make an educated guess to find which of the two buildings the RiskScape attribute were pertaining to, no satisfying automated approach could be developed to merge these instances. This case was thus discarded.

More details can be found in section 5.6.6 Properties with two street addresses and one or multiple RiskScape

Instances of the thesis (Roeslin, 2021). Section 5.6.6 also includes multiple figures showing examples of properties having two LINZ NZ street addresses per polygon. To keep this paper to a reasonable length such figures were not included.

Thanks for the comment.

Re: action for 2 LINZ points and 1 RiskScape

The wordiness has been removed. In short, those points were discarded.

Re: Percentage to Table 1

Thanks for the suggestion. The percentages have been added in brackets for each LINZ and RiskScape scenario of Table 1. Out of the selected instances in Christchurch, 89% have 1 street address point per LINZ property title, 7% two addresses per property, and 4% have three instances or more. For the RiskScape data, after merging in ArcMap it was found that 29.3% of the properties have one RiskScape instance, 43.6% have two instances, and 27.1% have three RiskScape points or more. 

Thanks for the comment.

The seismic demand captured through the peak ground acceleration (PGA) was interpolated using the inverse distance weighted (IDW) technique, applying the IDW spatial analyst function in ArcMap (Esri, 2021). 

In the paper, the explanation on how the seismic demand was interpolated from the ground motion recordings obtained from GeoNet is explained in section 3.3. 

Thanks for pointing that out. 

The EQC data entails a feature called ‘EQC Building Sum Insured’. As EQC provided a maximum cover of NZ$100,000 (+ GST) at the time of the Canterbury Earthquake Sequence, it was expected that all the individual dwellings for which one or multiple claims have been lodged during the CES would have an EQC Building Sum Insured equal to NZ$115,000. However, the initial exploratory data analysis (EDA) revealed that for some of the instances the building sum insured was not at NZ$115,000 (see Figure vi). As only a few instances were not equal to NZ$115,000 (2,594 instances for 4 Sep 2010 and 2,329 instances for 22 Feb 2011), it was decided not to include those instances (in the objective of retaining accurate data for the training of the ML model).

Figure iv: Number of instances in the attribute EQC Building Sum Insured (categorised to simplify the visualisation)

Thanks for the remark.

As mentioned in the reply to comment #1, the selection of the training, validation, and test set has been revised. All of those sets are now coming from the same earthquake event. For new earthquakes, a sample of buildings will be selected and added to the training set. Figure 10 has been updated to reflect those changes.

Thanks for the comment.

The min-max scaling of the numerical features was performed using the sklearn.preprocessing.MinMaxScaler available in scikit-learn (Pedregosa et al., 2011). A pipeline containing the min-max scaler was created. All sets were passed through the same pipeline.

Thanks for pointing that out. 

The limitations mentioned in this section refer to the overall limited model performance not to the limitations of the random forest algorithm itself. The first section of the phrase has been rewritten to clarify that it is related to the overall model accuracy. 

Thanks for pointing that out.

We are currently having a deeper look at the retraining of the SVM model. 

Thanks for the comment.

Sorry for the confusion. The paragraph has been rewritten and Fig 10 has been improved to clarify that the training, validation, and testing are initially done on one earthquake only.

Thanks for the remark.

The ML model currently presented in this study is categorical. Yet, the target attribute BuildingPaid is initially numerical but soft-capped at NZ$100,000 (+GST) or NZ$115,000. Having the actual numerical distribution of the extent of the claims might enable a deeper analysis of the target attribute and can possibly enable better performance for a regression model. This might alleviate the need to transform BuildingPaid from a numerical attribute to a categorical feature. 

Thanks for the comment.

The selection of the training, validation, and test set has been revised. The updated version is schematically shown in Fig 10. The training and the test set are now sourced from the same earthquake. The validation set is implemented using k-fold cross-validation as part of the hyperparameter tuning. Four models have been trained and tested using data from the four main earthquakes in the CES (4 Sep 2010, 22 Feb 2011, 13 Jun 2011, 23 Dec 2011). 

Figure i: New Figure 10

Thank you for the suggestion.

As mentioned in the reply to comment #1, the process of selecting the training, validation, and test set has been reviewed. The training and test set are now clearly separated thus ensuring that no data leakage can occur. Additionally, the way the model should be applied to future earthquakes has been clarified in Fig 10 as well as in the paper.

We are here deliberately focusing on the claim data related to the four main earthquakes rather than the claim data pertaining to the aftershocks. Figure ii shows the situation in the area around Christchurch as of December 2012 (one year after the end of the CES). It can clearly be seen that the earthquakes with the larger magnitudes were: 4 Sep 2010, 22 Feb 2011, 13 Jun 2011, 23 Dec 2011. Severe buildings damage, and hence higher claims, are mainly related to those four earthquakes as the seismic intensity and liquefaction occurrence were larger for those events, hence the selection of those earthquakes in this study.