Türkiye is in a region that is prone to natural hazards, where a large-scale disaster happens every 7 to 8 years . Among the different types of disasters, earthquakes are responsible for the most extensive losses in terms of both human life and property, accounting for 60 % of disaster-related fatalities in Türkiye . Following earthquakes, landslides (which mostly take the form of rock falls, slides or flows, or mass movements), floods, snow avalanches, and large-scale wildfires are amongst the most commonly occurring hazardous events that have adverse impacts on human lives, as well as the environment and the economy . Our case study area Istanbul city is also prone to hazardous events, such as earthquakes, flooding, landslides, tsunamis, and extreme weather events . However, our site selection is not only related to Istanbul's location in a hazard-prone area but also mostly related to its high population density and high level of economic investments that increase the expected losses from possible hazards in the city. Istanbul is the 15th most populated city in the world, with a population of approximately 16 million, and it is also the largest metropolitan city in Türkiye . After the 1930s, the city of Istanbul grew steadily and became the heart of Türkiye's economy, producing almost 31 % of the national GDP in 2021 . In the last century, the economic growth triggering mass migration to the city induced uncontrolled illegal housing with low-quality building materials in hazardous areas . Additionally, building codes were updated in 1997, and before that, even if legally constructed, buildings were built with less stringent building codes which did not consider disaster risk . This rapid and uncontrolled urban growth increased vulnerability to hazards in the city . Hence, our study area is selected as a suitable setting for our research on social vulnerability because it is a hazard-prone zone with high population density and poor-quality housing.

In this study, we relied on the pre-constructed SoVI as an indication of the social vulnerability of the households. By that, we used SoVI as the outcome of the machine learning models we tested. The SoVI score does not have any unit, and, rather than its absolute value, its importance lies within its comparative value across various households . Various authors dichotomised social vulnerability index scores in their research both for ease of interpretation and to identify those most vulnerable . In this research, we also aimed to discriminate between the most vulnerable households and all others. Therefore, we defined households with high social vulnerability (SV) as those with SoVI scores +1 standard deviation from the mean, which corresponds to 17.2 % of the households, whereas the rest of the households were deemed as low SV. Thus, a binary variable (with an approximate imbalance ratio of 1/5 in favour of low SV) was generated as an indication of social vulnerability level, which in turn was used as the primary outcome for all the further analyses presented in this paper. Further, from the statistical point of view, we preferred to dichotomise the outcome rather than using it as a multi-category variable, as the available performance metrics for a multi-class confusion matrix are limited compared to a binary classification problem, and the complexity of analysis increases with the increase in a number of classes . Therefore, in accordance with our motivation and for interpretive reasons we used SoVI as a binary outcome.

We have restricted the variables that are used in the ML models as input variables to quantifiable predictors, which can be obtained from various institutional databases without requiring a household-based survey that is costly and time intensive. These quantifiable predictors are related to the socio-demography and socio-economy of the households as well as housing information. The list of institutions to which the variables used in this study are related is given in Sect. S2. Here we note that, although the household data used in the to construct SoVI are focused on earthquakes, the indicators used for social vulnerability classification in the present study can be implemented in a more generic way to assess the possible impact of social vulnerability to other hazards.

Prior to model development, the predictors were prepared in terms of data representation, standardisation, and feature selection. As the predictors represent household characteristics, they were sought at the household level. As stated by , data representation is about enabling better interpretation of the relevant information. Therefore, the predictors which are measured at the household level, such as the number of women, men, <5 year olds, >65 year olds, and income earners were taken in proportion to the given household's size (HhS). Then, in order to make the variation of continuous variables comparable, these variables were standardised into the same scale with unit variance standardisation . For the final step, we used feature selection prior to processing the data, and we identified the predictors with near-zero variance, as the predictors which take only one value may cause numerical problems during resampling . The set of 26 variables used for model building is presented in Table , along with their relevance in relation to the objectives of our study.

We developed models for the classification of households in terms of their social vulnerability in the event of an earthquake using six supervised machine learning algorithms: classification and regression tree (CART), random forest (RF), artificial neural network (ANN), support vector machine (SVM), naïve Bayes (NB), and k-nearest neighbours (KNNs). The predictive performances of these ML models are compared to that of the logistic regression (LR) model, which is a traditional statistical technique used for binary classification. Supervised ML adopts an algorithm to learn the mapping function from the input variables to the output variable, and it is well suited to classification problems. Models were developed using the variable set in Table  as the input variables, while a binary indicator of the social vulnerability level of each household was the output variable. We developed a prediction model using 90 % of the dataset to train the underlying algorithm, while 10 % was held back as independent testing data for evaluating the performance of the models. We note that these algorithms have different tuning parameters. For different tuning parameter alternatives, the choice of the optimal tuning parameter was determined by the largest area under the curve (AUC) value of the receiver operating characteristic (ROC) curve using the automated grid search. The details regarding the machine learning models and R software packages used for the analysis are provided in Sect. S3. The workflow for the model building is shown in Fig. .

The characteristics of the study population were summarised using descriptive statistics. Pearson's chi-square tests were used to compare categorical variables, and independent samples t tests or non-parametric Mann–Whitney U tests were used to compare continuous variables between the high and low SV groups depending on the data distribution. In studies with large sample sizes, in addition to p values, it is also relevant to provide effect sizes, as it can help decide whether the difference found is meaningful or not . Thus, we have reported effect sizes in the univariate comparisons that measure the strength of the relationship between two variables along with the p values to assess whether the effect of a variable is real and large enough to be useful or not. Cohen's d statistic with sample size adjustment was used for normally distributed continuous variables, Cohen's r value, which is calculated by dividing the z value obtained from the Mann–Whitney test by the square root of the sample size, was used for non-normally distributed variables, and Cramér's V is used for categorical variables .

For various machine learning applications, confusion matrices were generated. Sensitivity, specificity, and accuracy with 95 % confidence intervals (CIs) were calculated for LR and each ML algorithms using different resampling and subsampling techniques. The models were fitted with two different resampling strategies and eight subsampling techniques. In addition, we fitted the models to the raw data without any subsampling, and thus we obtained results for 18 combinations of various sampling strategies for each ML algorithm.

In line with the objective of the study, we compared the methods in terms of their success in identifying the households with high social vulnerability, which is the minority class with a smaller prevalence in our study. Therefore, we used sensitivity (true positives/(true positives+false negatives)) as the primary measure for assessing the model performance. As an indication of model accuracy, we used balanced accuracy ((sensitivity+specificity)/2), which performs better on imbalanced datasets. We identified the best-performing method as the one with the highest sensitivity and balanced accuracy, provided that the AUC of the ROC curve is greater than 0.7, and the model could be considered acceptable to discriminate households with high SV from those with low SV .

The sensitivity and specificity of the best-performing method with those of other methods were compared with pairwise comparisons using McNemar's chi-square test . In addition, AUC comparisons were performed using DeLong chi-square statistics . Bonferroni adjustment was applied in these pairwise comparisons of ML methods, and α<0.05/7=0.007 was considered as an indication of a statistically significant difference in terms of performance metrics between two methods.

As the final step of our analysis, the important variables of each model were assessed. Analysing variable importance is important in machine learning applications because it assists in the interpretation of the model. It can be performed in two ways: (1) by using a model-based approach which computes the contribution of the predictor variables to the model or (2) by evaluating the importance of predictors individually by conducting an ROC curve analysis for each predictor in turn . How to choose which approach to use depends on which ML model was employed.

Logistic regression models rank the variables according to standardised coefficients. The regression coefficients of continuous variables are standardised by dividing each coefficient by a value twice its standard deviation, as explained in . The coefficients for factor variables are left unchanged. The relative importance of the independent variables for ANN models are computed by Garson weights , which identify all weighted connections between the nodes of interest. In this context, the weights connecting the variables can be thought of as similar to coefficients in a regression model and are used to describe the relationships between outcome and predictor variables. In random forests, variable importance analysis is based on the prediction accuracy of the model. The average differences between the out-of-bag errors before and after permuting each predictor variable over all trees are calculated as an indication of the importance of a variable. The underlying idea is that a permutation of an important variable reduces the accuracy of the model more strongly than a permutation of an unimportant variable . On the other hand, another tree-based method, CART, does not use the permutation technique for measuring variable importance, as it is trained on a single decision tree. Instead, CART depends on an impurity metric – which is often called the “Gini-index” – for determining the importance of a variable when the outcome is categorical .

For classification models (e.g. NB, KNN, and SVM) there is no available model-specific variable importance metric. Rather, these models calculate the area under the ROC curve for each predictor variable, and this AUC statistic is considered as the measure of variable importance .

An open-access R Shiny web application was created for visualising summary statistics and predictive performances of the LR and ML methods for the classification of households in terms of their social vulnerability level. Users are able to examine the distribution of the characteristics of the households with high and low social vulnerability, compare the performances of ML and subsampling methods based on user-defined evaluation criteria, assess variable importance rankings for each ML method, and obtain the area-based calculations of the variables on the Istanbul map. The R Shiny web application is freely available online and can be accessed at https://oyakalaycioglu.shinyapps.io/Social_Vulnerability/ (last access: 13 June 2023). The components of this R Shiny application are presented in detail in Fig. . All analyses were performed in the statistical programming environment R version 4.0.3 , and the machine learning model development was carried out using the R caret package . The spatial distribution of the important predictors within the city scale was expressed via the 3.10 version of the QGIS software .

The prevalence of households with high social vulnerability to a possible hazard in Istanbul was 7052 (17.2 %) among 41 093 households. The median household size was 3, with values ranging from 1 to 14 residents, and the median average age of the households varied between 8.8 to 85 years with the median being 35.5. The median of the average education was 8 years (range: 0–17 years) in the entire survey sample, while it was 8.8 years (range: 0–17 years) in those households with low SV and 6 (range: 0–16.3 years) in those households with high SV. Additional comparisons between social vulnerability levels in terms of socio-demographic, health, and socioeconomic information are demonstrated in Table . Households with high SV were often overcrowded, less educated, older, had a low number of income earners, had low levels of savings, and had less access to social security and health insurance compared to the low SV group. The statistically significant variable with the largest effect on social vulnerability was the average education of the household (Cohen's d=0.947), followed by the ratio of income earners (Cohen's d=0.366) and the ratio of over 65 year olds in the household (Cohen's r=0.120), having social security (Cramér's V=0.211), having health security or insurance (Cramér's V=0.226), having natural gas heating at home (Cramér's V=0.152), the presence of anyone with a disability or who is elderly and needs care at home (Cramér's V=0.142), and having savings for emergency situations (Cramér's V=0.135).

The comparison of the machine learning models in terms of their sensitivity, specificity, balanced accuracy, and AUC for different subsampling methods are presented in Fig. . The additional comparisons of models using other evaluation metrics (e.g. positive prediction value, negative prediction value, accuracy, F1 score, etc.) can be found in the R Shiny application. Within these comparisons, no substantial differences were observed in the model performance indicators of LR and different ML strategies between RCV and bootstrap resampling methods. Therefore, we present the results that were obtained with repeated 5-fold cross validation.

As mentioned earlier, the dataset suffered from imbalanced class variables, particularly the outcome variable, and as such significant differences were observed when subsampling strategies were applied. Using the standard algorithm without subsampling (referred to as “Original”) resulted in poor sensitivity (Fig. a), and inflated specificity (Fig. b) rates, due to the class imbalance in the studied sample where the negative class is dominant. Based on the criteria that AUC>0.7, overall, the methods fitted with under subsampling inside the resampling procedure (referred as under(in)) performed better in terms of model performance metrics when compared to other subsampling methods. The highest balanced accuracy for each method was also obtained with under(in) subsampling (Fig. c).

In Table , all ML methods using under(in) subsampling were compared to their counterpart using the original data without imbalanced subsampling. Here we remind the reader that the priority in this study was to assess the performance of the models in terms of their success in identifying the households with high social vulnerability, which is the minority class but therefore also the positive class. Using the under(in) subsampling strategy demonstrated superior sensitivity and balanced accuracy rates compared to using original data and other subsampling strategies. Therefore, the results obtained with under(in) subsampling are considered for further comparisons between ML methods. Classification results for the ML models using under(in) subsampling are presented with ROC curves in Fig. d. The ROC curves for all other subsampling strategies with all other methods can be found in the R Shiny web application.

The best-performing method in terms of AUC, accuracy, balanced accuracy, and sensitivity was the artificial neural network using the under(in) subsampling strategy (AUC: 0.813 (0.800–0.826), accuracy: 0.724 (0.710–0.737), balanced accuracy: 0.730 (0.790–0.752), sensitivity: 0.740 (0.706–0.772), specificity: 0.720 (0.705–0.735)). Naïve Bayes (NB) also produced a high sensitivity rate of 0.871 (0.843–0.894); however, it resulted in significantly lower specificity (0.502 (0.485–0.519)) and overall accuracy 0.566 (0.550–0.581) compared to ANN (p=0.003 and p<0.001, respectively). While ANN balances sensitivity (0.740) and specificity (0.720), NB emphasises sensitivity (0.871) over specificity (0.502). All other methods using under(in) sampling provided similar sensitivity rates between the range of 71.9 % and 72.9 % and specificity rates between 69.9 % and 72.4 %. When AUC was considered, CART was also significantly worse than ANN (0.782 (0.768–0.796) vs. 0.813 (0.800–0.826), p=0.005). Logistic regression, random forest, support vector machine, and k-nearest neighbours did not show significant differences from ANN in terms of performance metrics.

In Fig. , a visual summary of variable importance analysis is presented as the relative importance of the predictors, as indicated by the ML methods using under(in) sampling. As the methodologies used for analysing variable importance vary across different models, we averaged the variable importance rankings obtained with all models in Fig. a. The most important variable for every model is given a score of 100 %, followed by the next important variable which takes a relative value between 0 and 100. The variables which appeared in the top 10 most influential variables in all seven models were education, having social security, the ratio of income earners in the household, and having savings for emergency situations. Of these variables, the variable with the highest average importance was education.

In Fig. b we investigated the relative importance of the independent variables within the top-performing model, ANN under(in), using the approach suggested by . Based on this model, the most important variable for the classification of households’ social vulnerability appeared to be having social security. The other predictors with over 50 % of relative importance were a mixture of demographic and economic variables including living in a squatter house, job insecurity, ratio of the over 65 year olds in the household, owning a house outside of Istanbul, household size, the ratio of income earners in the household and having savings for emergency situations.

Based on the variable importance analysis with the top-performing model, ANN under(in), we performed area-based calculations to compare the neighbourhood characteristics in Istanbul. For categorical variables, the prevalence in the neighbourhood was calculated, while neighbourhood averages were used for the continuous variables. The three most important predictors of social vulnerability level were subsequently displayed as a five-category map in Fig. .

For Fig. a, the areas represented with dark red colours, below 70 %, indicate those neighbourhoods with the lowest social security, and these areas are prevalent in the outer regions of the metropolitan area. On the other hand, those neighbourhoods close to the central region mostly cover households with a higher prevalence of social security benefits. The number of neighbourhoods with a high density of squatter housing (>20 %) was 27 (Fig. b). These neighbourhoods are scattered throughout the city and are not concentrated in any specific region. The households with job insecurity are mainly located in the central region of the city (Fig. c). The distribution of all other variables across neighbourhoods of Istanbul can be found in the R Shiny web application.

In this study, we demonstrated that it is possible to predict the social vulnerability of households with a certain degree of precision using household indicators available within the databases of various institutions and public authorities. Based on our results, the best-performing ML method for identifying households with high social vulnerability was ANN using under subsampling within the resampling procedure to address the problem of class imbalance (AUC = 0.813, balanced accuracy is 73 %, sensitivity is 74 %, and specificity is 72 %). ANN is often considered an effective and useful tool for identifying hidden relationships between socio-demographic and socio-economic variables that arise in social science research . This may imply that the interrelated social relations between the variables in our dataset may be best handled by ANN. Apart from CART and NB, all methods provided similar AUC results (0.80) with no significant differences. There was no significant difference between the ML methods, except NB, in terms of the performance of identifying households with high social vulnerability (i.e. sensitivity).

A model with an AUC greater than 0.80 was considered to have an excellent discriminative ability by . Therefore, our proposed ANN model, with AUC of 0.813, indicated a good ability to discriminate households with high social vulnerability in a hazard event in Istanbul from those with low social vulnerability. Similarly, the AUC values achieved with RF and KNN were greater than 0.8. In terms of predictive accuracy, we obtained the largest balanced accuracy (73 %) with ANN. Further, the accuracy obtained with ANN and other models did not differ significantly. We considered the accuracy of our optimal ANN model to be acceptable, as the value is halfway between 50 %, which is useless, and 100 %, which is perfect .

A limited number of studies have used ML to predict hazard-related social vulnerability and reported performance metrics. achieved an AUC of 0.780 using the CART model to predict the social vulnerability of residential units in Andalusia with dwelling variables. Similarly, we obtained an AUC of 0.782 with the CART model when under sampling was used. When demographic and social indicators were used with an ANN model, obtained a balanced accuracy of 86.1 %. reported a high accuracy of 95.6 % with ANN using regional indicators when predicting the social vulnerability of municipal zones in Tabriz, Iran. Compared to these studies, we obtained a relatively low accuracy with our ML models, as we focused on proposing an optimal modelling strategy using readily available household variables. Thus, our modelling approach can be useful for decision makers to take immediate action for the most vulnerable households, and there is no doubt that the predictive performance of our models would benefit from incorporating more predictor variables.

An important aspect of our study was to find the most viable solution for the imbalance problem in our dataset, as the imbalance ratio between the high and low SV groups was around 1/5. When no subsampling strategy was applied to handle imbalance problem, we obtained poor sensitivity rates. A 39.3 % to 61.7 % gain in sensitivity was achieved with different ML models when under(in) subsampling was applied, and therefore the imbalance was being addressed, compared to using the original raw data without subsampling.

In our study, when ML models without subsampling strategies were used, the overall accuracy was higher due to the inflated specificity compared to the models using subsampling strategies. The standard application of ML model targets is to maximise the overall accuracy. Therefore, if they are trained on imbalanced data without considering imbalanced classes, they tend to over predict the class with higher frequency , which is the low vulnerability group in our dataset. This increases specificity and therefore reduces sensitivity. Therefore, the models based on the original imbalanced data resulted in lower sensitivity and failed to identify households with high social vulnerability, and they failed to meet our aims in the study.

Among subsampling methods, the random-majority under-sampling approach resulted in the best performance for all ML methods. This method discards data points from the majority class (i.e. low vulnerability group) at random until a more balanced distribution is reached, while training the models. Our dataset was sufficiently large to not be negatively affected by the discarding of data. Our results obtained with random under sampling are consistent with the ML literature, in the sense that if the size of the dataset is large then it is better to employ an under-sampling method .

Variable importance rankings tended to differ depending on the technique employed. Therefore, initially we aggregated the results of the variable importance analysis. On average, education was found to be the most important variable in all methods, followed by having social security, the ratio of income earners in the household, and having savings to be used in emergency situations. Within the top-performing model, ANN, the most important variable was found to be social security, followed by living in a squatter house, and job insecurity. When we discuss these results based on socio-urban conditions in Türkiye, we can easily comprehend that education and social security are interrelated factors, as more educated citizens tend to work in jobs with social security. Second, income and savings represent households’ economic power to cope with hazards.

Social security refers to the right to have the guarantee of unemployment benefits, retirement pensions, public protection from job injuries, and access to public health coverage, gained through regular work and employment . The lack of social security and insurance, particularly in a demonstrably unstable economy, increases vulnerability to many kinds of crises, including disasters and health emergencies such as pandemics. In our research, having social security actually means being able to get different kinds of socio-economic and health support in sudden shocks, which also covers the aftermath of a hazard, as the individual is registered in the public health system. In Türkiye, the rate of unregistered labourers who are not affiliated with the Social Security Institution in total employment was 27.4 % , while most unregistered labourers were found in the agriculture and service sectors . Unregistered employment means that no social insurance premiums are paid by the employer; thus, employees cannot benefit from social security . However, people in agriculture are mostly self-employed and do not have social security because they cannot afford to pay social security premiums regularly. Hence, the map we have presented on the different social security status of neighbourhoods with respect to the household survey indicates the northwest of Istanbul as having lower social security, which may be due to a large number of agricultural areas in that region. However, those neighbourhoods close to the centre of the Istanbul metropolitan area are mostly inhabited by people employed in the services and industrial sectors, with a higher rate of registered employment and thus a higher prevalence of social security benefits. Moreover, in the data presented, the prevalence of social security in the high vulnerability group is around 72 %, whereas it is as high as 91 % in the households with low vulnerability.

Based on our findings, living in a squatter house was the second most important variable of social vulnerability using the ANN method. Squatter housing comprises houses that are assembled quickly and do not conform to the technical and legal standards (called “gecekondu”, as the Turkish name for poor squatter settlements). Hence, this type of housing represents at-high-risk buildings in the event of geological and climatic hazards and is more likely to be damaged in such events, which implies higher vulnerability to hazards. One of the large-scale hazardous events anticipated for Istanbul is an earthquake with a magnitude greater than 7 MW, which is predicted to strike the city within the next 30 years with 42 %–47 % probability . Previous studies inform that a large proportion of buildings in Istanbul, including squatter settlements, are not earthquake resistant . Furthermore, squatter housing is linked to a poor socio-economic household profile. It is known that poorer people are more vulnerable to natural hazards, as they settle in buildings that are at higher risk but more affordable to them because of cheap rents . In particular, squatter houses are very low-quality buildings, and when taken together with the poor socio-economic characteristics of their residents, they represent high social vulnerability for households. A study by in Andalusia, which used CART, showed the importance of dwelling variables on social vulnerability, such as the average age of constructions and the density of buildings in a particular district of an urban area. In our study, the age of the buildings was not available in the data; however, the type of housing was found to be an important predictor of social vulnerability.

With the ANN method, the third-highest-ranked variable was job insecurity. The spatial distribution of neighbourhoods in terms of job insecurity indicates that the centre of Istanbul close to the Marmara Sea is densely populated, with households with job insecurity representing the possible unemployment figures in those crowded areas. Further, as mentioned above in the social security indicator, the labour market opportunities in Türkiye are highly dominated by the casual or seasonal employment opportunities . Such forms of casual employment are highly fragile since the labourers are not in full employment and not registered in the social insurance system. A recent study showed that casual and unregistered employment increases social vulnerability to natural hazards . These may be either in the form of casual, seasonal employment or self-employment, where social security and social insurance registrations are not provided by the employers, and the employees could not afford to pay their premiums regularly by themselves. These types of employees and small businesses mostly fall below the poverty line even if they may be observed as working . Those households which depend on casual, unregistered employment and small businesses have a high probability of experiencing vulnerability when a disaster strikes, as they may experience loss of any economic means in that situation. There is an important difference between the job insecurity and social security variables. Job insecurity actually reflects the situation where the individual has no regular income; on the other hand, social security is covering all kinds of support and compensation mechanisms not only limited to the economic means of regular income. Although not limited to these, there might be several reasons for the difference between neighbourhoods in terms of these two variables. For example, it may be that, in the rural areas of northwest Istanbul, the individuals may not have social security, but they own their land and small businesses, and their jobs are more secure even though they may have a limited income . In contrast, in the centre of the city most of the population is in wage employment, where a major group is in regular registered employment beside a significant group of the unemployed or those working on a daily basis in casual jobs . Hence, unemployed or those in daily jobs may suffer job insecurity and a high risk of losing employment and/or income if caught by a hazard. Moreover, the individuals working in the service sector, which is common in Istanbul neighbourhoods, may suffer more from the possibility of work closures after a major hazard. For example, during the COVID-19 pandemic, when small workplaces have been required to close or restrict their services for a long period of time, most working people suffered severe job and income losses; hence, high vulnerability emerged . While Istanbul took a 41.9 % share of the total services sector in Türkiye in 2021, the share of the services sector in Istanbul's total gross domestic product was 33.7 % .

The other variables among the top 10 most important predictors that contribute to the model performance of the ANN model were a mixture of demographic and economic variables. These included the ratio of over 65 year olds in the household, owning a house outside of Istanbul, household size, the ratio of income earners in the household, having savings for emergency situations, owning land outside of Istanbul, and the level of education of the residents. The demographic variable of having elderly (>65 years) people in the household being an important predictor of social vulnerability to hazards is also highlighted in the literature . High education which lowers social vulnerability is a factor that is related to both having social security, as mentioned before, and an increase of awareness of taking precautions for possible hazards. The other significant variables like having property and savings are both related to income, where property outside the city may give more chances for the households to have a safe shelter after a major hazard. Furthermore, the associations between income and level of education are strong and consistent; that is, children from poorer family backgrounds have a tendency towards achieving a lower level of education . Also, the poor have less access to resources which may be effective in reducing risks, such as extra savings for preparing their houses for a hazard or accessing risk preparation information, and therefore cannot take as many precautions to cope with a disaster when it occurs .