Debris flows are among the most hazardous geological phenomena, which directly threat human lives in the light of their high energy and rapid propagation over slopes and drainage systems. In order to predict these phenomena, together with physically based approaches, which are mainly focused on the detection of the rainfall thresholds responsible for their triggering (e.g. Peres and Cancelliere, 2014; Bordoni et al., 2015) and on the physical modelling of the propagation phase (e.g. Schraml et al., 2015), susceptibility models (Brabb, 1984), suitable to depict prediction images of the sites where these phenomena are more likely to activate on a catchment/regional scale, are required as well. Combining the two approaches allows optimization of the use of early warning systems (e.g. Lagomarsino et al., 2015; Segoni et al., 2015; Stähli et al., 2015) – in doing so mitigating the debris flow risk on a catchment/regional scale.

Landslide susceptibility assessment can be achieved by means of different methods, among which the stochastic approach has gained increasing importance in the last 2 decades in regional assessment applications. In fact, statistic models produce objective, quantitative and verifiable estimates of the spatial probability for new landslides in a given study area. Moreover, the stochastic approach is very easily implementable on geographic informative systems (GIS), making use of the very diffused nature of present databases of physical–environmental attribute layers. These methods are based on some generally accepted assumptions, the basic one being the past is the key to the future (Carrara et al., 1995). Therefore, a susceptibility model constructed to reproduce a past known landslide spatial distribution, will also be able to predict the future locations of new failures. In particular, for a given study area, statistical techniques allow the derivation and testing of the multivariate relationships between the spatial distributions of an inventory of landslides (the known target pattern) for significance as well as testing a set of physical–environmental variables (the predictors), which, acting as controlling factors, are supposed to drive the slope failures, on the basis of a geomorphological model. In the framework of the above-recalled principle, the new landslides (the outcomes) will occur under the same conditions which explain the known landslide distribution. Thus, a calibrated predictive model optimizes the functional relations between predictors and outcomes, maximizes its skill in fitting the known target pattern (the calibration data set), and it is finally tested for its correct reproduction of the unknown target pattern (the validation data set). As the controlling factors are selected among the time-invariant preparatory causes, regardless of how old the landslide inventory employed to calibrate the model is, as far as the basic assumption holds, any calibrated model will be able to predict any past or future unknown target pattern.

Unfortunately, very often, susceptibility assessment studies are affected by a lack of temporal information on the landslide inventory, which makes it impossible to perform a pure temporal or chrono-validation.

Based on the scheme described above, in order to elude the lack of temporal information, strategies for the validation of the predictive models can be defined. Specifically, when seasonal or event inventories (Guzzetti et al., 2012) are not available, a validation can be performed by following a random time partition procedure (Chung and Fabbri, 2003). In this case, the source inventory is split into a calibration and a validation subset to simulate the known and the unknown target patterns, respectively. In this work, the above scheme is defined as a self-validation procedure to highlight the notion that, under a morphodynamic perspective, calibration and validation patterns are actually two partial and complementary sides of the same event. Conversely, the term chrono-validation will be used when referring to pure temporal verification (Guzzetti et al., 2005), i.e. when the training and the test target patterns belong to two temporally separated data sets. A third scheme, frequently adopted for model spatial transferability or exportation (e.g. Von Ruette et al., 2011; Costanzo et al., 2012a; Lombardo et al., 2014; Petschko et al., 2014), is based on the adoption of two different catchments or areas for calibration and validation (spatial partition).

It is evident how the whole scheme of the stochastic approach is strictly dependent on the basic assumption being held. Any changes in the real relationships between preparatory causes and landslide activity will affect the prediction skill of the obtained susceptibility models. Extreme events produce morphodynamic responses that can lie outside of the general rule. In fact, due to intense triggering, such as a storm, the same area can result in an “out-of-range” slope response because it could not be correctly predicted by a model skilled in fitting “normal” landslide scenarios. This could be a result of the non-linearity of the relationship between preparatory causes and landslides in the domain of the trigger intensity. Besides, the Mediterranean storms are typically affected by randomness in time recurrence and magnitude and a great spatial variability, even at the scale of small catchments. It is therefore necessary to check for this kind of behaviour to find a strategy which maximizes the ability of a susceptibility model to predict extreme events.

In spite of the wide diffusion of landslide susceptibility studies by means of statistical modelling, few cases are focused on detecting predictive limits when facing storm-triggered multiple debris flow events (e.g. Von Ruette et al., 2011; Tseng et al., 2015). In particular, the application of specific validation strategies to evaluate the effect of the trigger phenomena in modifying the predictive performance of the models is very rare. A contribution to this topic is presented here, drawing on a case study in north-eastern Sicily, where two recent storm events (2007 and 2009) hit the Ionian side of the Peloritani Mountains (Fig. 1) with different intensities. Specifically, the study area is the Itala catchment (nearly 10 km2), which is located in the southern sector of the Peloritan ridge.

In order to investigate our topic, the debris flows activated on the occasion of the two extreme events were mapped by integrating remote and field surveys, and a simple set of predictors was prepared by utilizing a 1:50 000 scale geological map and a 2 m cell digital elevation model (DEM). Statistical models were obtained by applying the stepwise (forward) binary logistic regression technique (Hosmer and Lemeshow, 2000), which has been largely adopted in landslide susceptibility studies (Atkinson et al., 1998; Ohlmacher and Davis, 2003; Süzen and Doyuran, 2004; Brenning, 2005; Carrara et al., 2008; Costanzo et al., 2014; Lombardo et al., 2014; Heckmann et al., 2014), demonstrating suitability for the geomorphological task and producing high performances, also in comparative studies (Guzzetti et al., 2006; Othman et al., 2015; Rossi et al., 2010). Multi-temporal high-resolution images (provided by ARTA – Assessorato Regionale Territorio e Ambiente) were made use of in order to prepare two landslide event inventories (Guzzetti et al., 2012), so that two types of modelling procedure are performed and validated: self-validation, based on the random partition into a calibration and a validation subset of each event inventory, and chrono-validation, based on the temporal partition into the 2007 and 2009 cases. The latter procedure was applied to analyse the performances both of the 2007 calibrated model in predicting the 2009 debris flows source areas (forward chrono-validation) and of the 2009 calibrated model in predicting the 2007 debris flow source areas (backward chrono-validation). By analysing and comparing the predictive performances of binary logistic regression for the four types of models, the role of the triggering rainfall intensities is outlined and discussed.

Testing a susceptibility model against future landslides is quite a hard task, especially because it would require researchers to “wait for the future to happen” (Guzzetti, 2005). Nevertheless, when a multi-temporal landslide inventory is available, the validation can be performed using a temporal criterion to separate calibration and validation data sets. Among others, Guzzetti et al. (2005) performed a “temporal verification procedure” which evaluates the effect of five landslide inventory updates on the performance of a susceptibility model. Similarly, other authors used a temporal criterion to validate the results of landslide susceptibility analysis at different scales (Zêzere et al., 2004; Vergari et al., 2011; Wang et al., 2014), but none of them worked with storm-triggered debris flows and event inventories. Von Ruette et al. (2014) adopted a spatial partition scheme, with a partial insight into temporal validation which was limited for predicting the landslides triggered by two rainfall events in two close, but different, catchments. Chang et al. (2014) concentrated their focus on exploring the role of rainfall in controlling the chrono-validation performance for a much larger-scale case, demonstrated in a larger area (2868 km2), where a network of 24 rain gauges recorded nine great typhoon events.

The Messina area (Fig. 1) and the debris flow event of 2009 have been the focus of study in several scientific articles centred on different topics. Several studies have been devoted to the implementation of remote and semi-automatic techniques for landslide recognition and mapping of such a significant multiple occurring regional landslide event (Ardizzone et al., 2012; Mondini et al., 2011; Ciampalini et al., 2015). Del Ventisette et al. (2012) focused their research on the Giampilieri village area, analysing the triggering mechanism and estimating the volumes involved in the debris flow. They also applied a method based on conditional analysis in order to obtain a susceptibility map. Goswami et al. (2011) and De Guidi and Scudero (2013) explored the relationship between tectonic setting and landslide susceptibility, taking the Giampilieri and Scaletta catchments as study areas. Reichenbach et al. (2014) evaluated the influence of land-use change on debris flow susceptibility for the Briga catchment. Stancanelli and Foti (2015) compared two different numerical models for simulating the 2009 debris flow event in the lower coastal sector of the affected area. Aronica et al. (2012a) published a detailed description of the 2009 event, with an insight into the saturation conditions of the soils and an evaluation of the difference of DEMs in the total volume of mobilized material for the Giampilieri catchment. Rainfall thresholds for the landslide activations have been investigated by Gariano et al. (2015), in the framework of a regional study, and by Peres and Cancelliere (2014), who conducted a specific study on the Ionian-Peloritan area, hit by the 2009 event. Lombardo et al. (2014) tested spatial exportation techniques for logistic regression-based susceptibility models, in the Briga and Giampilieri catchments.

With regards to the 2007 event, Aronica et al. (2012b) applied a physically based modelling tool to simulate the debris flows affecting a very small catchment, located 5 km south of the Itala stream.

In contrast to the above-mentioned research, in this paper, by studying two well split event inventories produced by two triggering events with different intensities, the relationship between controlling factors of triggers and morphodynamic responses are examined, and their effects on the predictive performance of stochastic susceptibility modelling are verified. Moreover, until now, no study has been published for the Itala catchment on the 2007 event, nor chrono-validated models and maps have been produced for the 2009 event.

The application of binary logistic regression (BLR) for landslide susceptibility assessment typically requires the following steps: the partition of the study area into mapping units, which are then characterized with respect to a set of potential predictors; the assignment of stability conditions to each mapping unit, based on its spatial relation with a set of known landslides (e.g. inclusion or intersection); the extraction of a balanced (stable/unstable) data set from the whole set of mapping units; the regression of the modelling function; and the verification of the performance of the model in correctly predicting stability conditions for each pixel, the latter defined on the basis of a set of unknown landslides.

This chapter describes the methods and the model building strategies which have been adopted to investigate the main research topic: exploring skills and limits in predicting the source areas of storm-triggered debris flows.

The results of the cross-validation procedures for the one hundred 2009 and 2007 self-validated models are presented in Tables 1 and 2. Generally, the 2009 models (Table 2) resulted in a better performing prediction with lower (0.336, for 2007; 0.219, for 2009) and more stable error rates. Similarly, the ROC-AUCs (Table 3) attested to the good quality of the models, with a higher performance for the 2009 model (2009 AUC was 0.85, 2007 AUC was 0.70) and no evidence of overfitting.

With regards to the predictors, the 2007 model suite selected five variables (Fig. 9), four of which had a frequency of more than 5/10: west and south-west slope aspect, steepness and phyllites to meta-arenites (FDNb) outcropping lithology resulted as the main causative factors for the 2007 debris flows. A larger set of variables (17) was included by BLR in the 2009 model suite (Fig. 10), 15 of which were selected more than five times. Among the topographic variables, the most important were: steepness, all the pixels without any northward aspect component, profile curvatures (both concave and convex) and plan convex curvature of slopes. Together with topographic variables, FDNb and paragneiss to mica shists (MLEa) lithologies, distance from tectonic elements (DFAULTS) and chestnut forests (CF) and pastures (P) land-use classes were always selected with high and stable rankings. Concerning the β-coefficients, only profile curvature concavity, the variables DFAULT and CF and P land uses showed negative values, indicating inverse correlation with the debris flow source areas.

Once the overall quality of the predictive performance of the 2007 and 2009 models was assessed, regressions were run for the 10 full (without splitting into calibration and validation subsets) data sets of each event inventory, which maximized the fitting of the models. For both these full self-validated models (Fig. 11), the obtained ROC-AUCs are above the good performing threshold (> 0.81 for 2007; > 0.87 for 2009), with average error rates of 0.26 for 2007, and 0.22 for 2009. The 2007 and 2009 full models were then submitted to forward and backward chrono-validation, respectively, resulting in largely acceptable ROC-AUCs (> 0.75) and error rates (< 0.3), although a loss in the predictive performance of both the temporal predictions was observed. In particular, by comparing the self-validation and the chrono-validation performances, a decrease in AUC from 0.81 to 0.77 for 2007, and from 0.87 to 0.78 for 2009, arose. Also, the mean error rate values increased from 0.26 to 0.30 for 2007, and from 0.20 to 0.28 for 2009. It is worth noting the strong decrease in performance affecting the 2009 model, which led the two chrono-validations to be almost equivalent. In Fig. 12, the calculated mean (over 100 replicates) ROC curves are shown. Coherently, the HK mean scores are comparable between forward and backward validations, presenting a maximum of 0.433 and 0.446, respectively (Table 3).

A spatial view of the obtained prediction images for the 2007 and 2009 models is given in Fig. 13. In particular, the susceptibility maps show the spatial distribution of the mean probabilities for the 10 replicates, whilst the error maps describe the dispersion of the estimates, represented by a 2σ interval.

At a first glance, the two susceptibility maps appear quite different: the 2007 map shows a more diffused and graduated susceptibility, with the north-western and south-eastern sectors of the catchment hosting high susceptible areas. On the contrary, the 2009 map is characterized by a marked spatial separation between the north-eastern high susceptible sector and the remaining larger part of the catchment, which has a low susceptibility. In terms of error maps, the 2007 model is affected by a generally higher level of error, with the maximum values located in the central sector and minimum values along the stream network. The 2009 model, on the contrary, produced lower errors, with the exception of the stream network, which is characterized by relatively higher values, and two single small areas, corresponding to the outcrops of poorly diffused lithologies (see Fig. 1).

To compare the two landslide susceptibility maps, taking into consideration the distribution of the debris flows which occurred in 2007 and 2009, LIPs were located onto a map of the residuals. This map represents the difference between the two (2007 and 2009) mean susceptibilities (Fig. 14). The residuals confirmed the dissimilarity between the two models in estimating the susceptibility of the catchment, with higher probabilities in the southern and north-western sectors for the forward-validated models, and in the north-eastern sector for the backward-validated models, respectively.

By comparing the two susceptibility estimates in a dispersion density plot (Fig. 15), the above-described trend is verified. The two models linearly agreed in the higher range of susceptibility, whilst a larger dispersion existed in the lower and intermediate susceptibility range. In particular, for the stable areas (near the origin of the plot) the higher densities pixels are shifted toward a more than 45∘ steep linear trend, marking an overestimation for the 2007 calibrated model.

From a binarized perspective, by setting the cut-off value for stable/unstable discrimination to 0.5, the final number of joint predictions (II, for TP, and IV, for FN sectors) was 77 %, whilst disjoint predictions (I and III sectors of the plot) reached 23 %. The two chrono-validated models performed in predicting the whole set of observed positives with different results: the backward-calibrated model produced 46 + 3 (67 %) true positives and 13 + 11 (33 %) false negatives for the 2007 LIPs, while the forward-calibrated model produced 395 + 50 (72 %) true positives and 90 + 81 (28 %) false negatives for the 2009 LIPs.

In this study, the findings of previous studies (Zêzere et al., 2004; Guzzetti et al., 2005; Vergari et al., 2011; Wang et al., 2013) regarding the effectiveness of temporal partition procedures to explain future landslides are generally confirmed here, even in the case of debris flows triggered by an extreme rainfall event. The above-described results attest to a symmetry between forward and backward chrono-validations, as well as the main assumption on which stochastic modelling is based. However, through the analysis of the self-validated models, it was identified that the 2009 model resulted in a higher predictive performance, with a higher number of selected variables. This could be interpreted as a direct consequence of the greater number of debris flows which compose the 2009 inventory (1 order of magnitude more), so that a larger spectrum of multivariate conditions of the slopes was involved in failures and included in the data sets for the fitting of the models. However, the first four selected predictors for the 2009 model correspond to those composing the structure of the 2007 model: slope morphology (steepness, curvature and aspect), soil use and outcropping lithology.

The comparison between the performances of the self-validated and the chrono-validated models has highlighted a loss in accuracy which is slightly more marked for the higher performing self-validated 2009 model. Therefore, although a large difference between the accuracy of the two self-validated models is observed, the comparison between the forward and backward chrono-validated models shows very smoothed differences in terms of ROC-AUC and error rates. This suggests that, in spite of the higher performance which the 2009 model obtained in classifying the same 2009 event, its skill in back-predicting the 2007 debris flow source areas is the same shown by the 2007 event in forward-predicting the debris flow source area of 2009.

The loss in performance demonstrated by the 2009 model suggests that using self-validated models for temporal prediction can mislead the user in estimating the performance of the model. In fact, one would expect that the model calibrated with the largest landslide inventory would be the best-performing in chrono-validation as it also includes the less extreme morphodynamic responses. However, in spite of the similar inner structure of the 2007 and 2009 models, the predictive performance of the 2009 backward model lowered to the same ROC-AUC and error rates of the 2007 forward model. The reason for this behaviour could be connected to the different local characteristics of the two storm events, which hit the slopes differently, even in such a small catchment. This would indicate, for this study case, that inside a 10 km2 area there are two different pasts and two different futures, depending on which of the two storm events are used for calibration. This is a similar finding to that obtained for chrono-validation procedures by Chang et al. (2014) in a larger-scale (tropical cyclones) study, whose predictive models even resulted in being “capable of predicting landslides triggered by a strong typhoon but not a weak typhoon” (i.e. their best model missed nearly all the landslide cells triggered by the weak typhoon events).

At the same time, a non-linearity of the morphodynamic response of the slopes (different coefficients and/or predictors) could affect the performance in chrono-validation: a larger event does not produce a larger response which include less intense storms, but rather a different one. The larger the difference between the triggering events, the greater the distinction in the response of the preparatory conditions.

In the domain of the predictors, this is highlighted by the different inner structures of the models. If compared to the 2007 event, the 2009 event also activated eastern and south-eastern-facing pixels, as well as high metamorphic-grade (MLEa) lithologies and terraced deposits; topographic curvatures, distance from faults and soil use (the latter with negative coefficients) have also taken an important role in controlling the distribution of the debris flow source areas. However, this richer structure of the model does not increase its predictive ability with respect to the distribution of the 2007 debris flows; the backward chrono-validation does not demonstrate this greater accuracy. This suggests that the 2007 debris flows were activated through different, even if largely overlapping, mechanisms.

In the domain of the geographical space, the map of the residuals provided a spatial view of the different behaviour of the two models, giving the interpreter clues for the real path followed by the two storm fronts inside the Itala catchment. The 2009 model markedly overestimated the susceptibilities in the central-northern sector of the catchment, whilst the 2007 model produced higher susceptibilities than 2009 in the north-western inner mountain sector. Regardless of the different intensities, this spatial trend suggests that the 2009 storm path was limited to the coastal area, whilst the 2007 storm affected the whole catchment more homogeneously, activating also the slopes of the mountain sector. This interpretation is also confirmed by the different spatial distribution of the debris flows of the two event inventories and it agrees with the findings on a much larger scale of Chang et al. (2014).

However, from a risk perspective, the difference between the two models did not produce a significant loss in prediction, as only a limited number of cases resulted in a false positive prediction. This is why the mapped debris flows are largely located in the more susceptible pixels. However, the results of the present research have confirmed that the larger difference between the two models has been observed in the intermediate susceptibilities interval, which is the same region of the error plots where the self-validated models show poor precision. This difference is also attested by the HK scores, which confirmed the good prediction skills, but with maximum values proximal to 0.45. Under the considered triggering conditions, the multivariate relationship between debris flow activation and predictors is in fact linear, so that no single marked cut-off value for probability accurately discriminates positives from negatives. Nevertheless, it is worth highlighting the selection of a 0.5 cut-off value, which resulted in a higher performance for the temporal prediction of the positive cases (forward chrono-validation) of the 2007 calibrated model.

Finally, it is worth comparing here the results obtained for chrono-validation (AUC was0.77 / 0.78), with the ones from Lombardo et al. (2014), which applied a spatial exportation scheme in two catchments very close to each other. In fact, a higher performance (AUC was 0.83) resulted for the prediction skill of the transferability procedure which was adopted there, by calibrating the model in the Briga catchment to predict the Giampilieri debris flows, using event inventories produced by the same 2009 storm-triggering event. Sharing the triggering event allows for a better predictive ability, in spite of the circumstance that, in a spatial partition scheme, the calibrated model is totally blind with respect to the validation area, in terms of the spatial combination of the predictors and the target pattern (the unknown debris flows).

The results obtained in this research confirmed that the basic assumption on which susceptibility modelling is based (the past is the key to the future) must be critically accepted in the case of extreme events. In fact, in the case of the two storm events considered here, the dissimilarities in the intensity and the real path of the two storm fronts produced measurable differences in the behaviour of the two derived predictive models, both in the domain of the predictors and in the spatial pattern of the susceptibility maps. Two main causes have been recognized here: on the one hand, the slopes did not linearly respond to the trigger intensity, so that different predictors and coefficients were fitted by the two regressed models; on the other hand, effects produced by the spatial non-homogeneity of the rain intensity for each single storm event, even at the scale of such small catchments, were detected.

In terms of the operative use of the susceptibility maps, the effects identified attest to the risk of either over- or under-estimating the susceptibility, both for the 2009 and 2007 models. In particular, limits arise in the general perspective of using the most severe and available inventory for calibrating the best-performing model. In fact, in this research it was verified that this best-performing self-validated model did not result in the most accurate one in chrono-validation, also demonstrating susceptibility underestimation and false negative production.

In the present study, the differences between the two models basically reside in the intermediate susceptibility interval, so that a precautionary approach in reclassifying the susceptibility map could be adopted, accepting the precision limits in the intermediate probability classes. However, larger differences between the triggering storms to which calibration and validation event inventories are connected could result in larger predictive limits and more misleading susceptibility maps.

The strict relation between trigger intensity, slope response and prediction performance arises also from the comparison of this study to another study carried out by applying spatial partition or transferability validation strategies in two adjacent catchments for the same 2009 trigger, obtaining a better predictive performance. In the opinion of the authors, this difference confirms limitations of the chrono-validation procedure when working with extreme rainfall events. For this reason, the application of transferability or chrono-validation should be evaluated from time to time on the basis of the availability of historical records of phenomena, information on the trigger event, and similarity with other areas where debris flow events have already occurred. At the same time, the production of susceptibility maps such as those presented in this paper constitutes a basic starting point for modelling propagation, run-out and magnitude associated to the predicted phenomena, so that an estimation of the debris flow hazard is achieved within a given area.