The predictive performance of single loss models and ensemble means is evaluated in terms of accuracy and systematic bias, using respectively the root mean squared error (RMSE) and the mean bias error (MBE). These are given by RMSE=1n∑i=1nX^i-Xi2 and MBE=1n∑i=1nX^i-Xi, where X^ is a vector of n predictions and X is the vector of observed values of flood loss.

The probabilistic skill of ensembles is evaluated using the continuous ranked probability score (CRPS), which is defined as the integrated squared difference between the cumulative distributions of predictions and observations (Weigel, 2012). We adopt the expression for the CRPS derived by Hersbach (2000), which is described as follows. Consider a set of n elements affected by a flood with corresponding observed losses x1,…,xn. Let there be m ensemble members, and let x^t,i be the prediction of loss given by ith ensemble member for the tth element, sorted in ascending order. Define x^t,0=-∞ and x^t,m+1=+∞. The CRPS is given by

CRPS=1n∑t=1n∑i=1mαt,iim2+∑i=0m-1βt,i1-im2, where αt,i=0ifxt≤x^t,ixt-x^t,iifx^t,i<xt≤x^t,i+1x^t,i+1-x^t,iifx^t,i+1<xt and βt,i=x^t,i+1-x^t,iifxt≤x^t,ix^t,i+1-xtifx^t,i<xt≤x^t,i+10ifx^t,i+1<xt.

The CRPS can be interpreted as an error measure, with lower values corresponding to higher probabilistic skill.

To assess ensemble reliability (i.e. whether ensemble predictions and observations are statistically indistinguishable), the rank histogram is adopted, which is constructed as follows. Consider an m-member ensemble prediction x^=(x^1,…,x^m) and a corresponding observation x. The rank of x in relation to the ensemble members of x^ is given by r=M+1, where M is the number of ensemble members that x exceeds (M≤m). For example, if x is smaller than all ensemble members, the observation has rank r=1, while if x exceeds all ensemble members, then r=m+1. If an ensemble is reliable, for a set of n prediction-observation pairs there should be n/(m+1) observations with each m+1 possible rank values, i.e. the histogram should be flat. Systematic deviations from flatness can indicate deficiencies in terms of ensemble dispersion and bias. Note that no ensemble is perfectly reliable, and random deviations from flatness are expected due to sampling uncertainty (Talagrand et al., 1998; Weigel, 2012).

Floods are a recurring natural hazard in the Mulde catchment (7400 km2) located in Saxony, Germany. In recent years, this area has been severely affected by the June 2013 and August 2002 floods (Engel, 2004; Merz et al., 2014). The latter was triggered by record-breaking precipitation amounts in the Ore Mountains, which form the headwaters of the Mulde River. At the Zinnwald-Georgenfeld station, operated by the German Weather Service, 312 mm of rainfall were recorded within 24 h (Ulbrich et al., 2003). The flood caused many dike breaches and resulted in considerable loss in 19 municipalities in the German state of Saxony along the Mulde (Fig. 1).

The data used for this application case are listed in Table 2, and the results of individual model applications in terms of error statistics are shown in Table 3. The flood extension and water depths were estimated through hydro-numeric simulations (Apel et al., 2009) and hydraulic transformation (Grabbert, 2006). Return periods of flood peak discharges were derived from annual maximum series of mean daily discharges by Elmer et al. (2010). For the estimation of contamination indicators, inundation durations, flow velocity indicators and precautionary measures indicators, computer aided telephone interviews with affected households have been used (Thieken et al., 2005). The average floor areas of residential buildings and average building values are based on official statistical data about total living area for different types of residential buildings per district, and standard construction costs per square metre gross floor area (Kleist et al., 2006). Asset values with a spatial resolution corresponding to the inundation map (i.e. 10 m × 10 m) have been derived by applying a binary disaggregation method and using the digital basic landscape model ATKIS as ancillary information (Wünsch et al., 2009). Residential building type composition and mean residential building quality per municipality were derived by Thieken et al. (2008) using geo-marketing data from INFAS GEOdaten GmbH from 2001. Flood losses to residential buildings have been documented by the Saxon Relief Bank on the municipality level (Saxon Relief Bank, personal communication, 2005) and amount to a total of EUR 240.6 million. For more details, see Kreibich et al. (2017).

From 31 October to 2 November 2010, the Veneto Region was affected by persistent rain, particularly in the pre-alpine and foothill areas, with accumulated rainfall exceeding 500 mm in some locations (Regione del Veneto, 2011a). This caused multiple rivers to overflow, resulting in floods that inundated an area of 140 km2 and had a considerable human and economic impact. Three people lost their lives and 3500 had to evacuate their homes. Flood losses to residential, commercial and public assets were estimated to be EUR 426 million. Caldogno, a municipality with a population of about 11 000 located in the province of Vicenza, was among the most affected, with reported losses to those sectors reaching EUR 25.7 million (Regione del Veneto, 2011b). In this study, we adopt it as the second application case (Fig. 2).

The data used for this application case are listed in Table 2, and the results of individual model applications in terms of error statistics are shown in Table 4. The inundation characteristics were estimated using a coupled 1-D/2-D model of the study area between the municipalities of Caldogno and Vicenza, and validated using data from sources such as aerial surveys and interviews with the local population. Building areas were derived from the cadastral map issued by the Veneto region. Building properties (i.e. building type, structural type, quality, number of floors, and year of construction) were assessed through direct surveys to each damaged building. Building values were estimated based on data from the Chamber of Commerce of Vicenza. Losses to residential buildings were provided by the municipality of Caldogno and amount to a total of EUR 7.55 million. These correspond to actual restoration costs that were collected and verified within the scope of the loss compensation process by the state. Further details can be found in Scorzini and Frank (2015).

The first challenge in constructing a multi-model ensemble to estimate flood loss for a certain future application is identifying models that are better suited to be participating members. One of the requirements for the construction of successful multi-model ensembles is that participating models are skilful; if a model is consistently worse than the others in terms of prediction quality, it should not be included (Hagedorn et al., 2005). Unfortunately, testing the level of skill of a model in predicting loss, for a certain type of asset and application setting, is often not possible. Such exercise would involve applying each candidate model to estimate loss for a past flood event with similar characteristics, and quantifying its performance based on past loss observations for the same assets. However, data required to perform such assessments are usually not available, as scarcity of data is still a major problem in the field of flood risk (Merz et al., 2010). Moreover, exposure and vulnerability tend to change over time, which is likely to affect loss estimates (Tanoue et al., 2016). Another issue of a more practical nature is that collecting, implementing and comparing flood loss models is laborious and time consuming. Because of the economic constraints that inevitably exist in any practical application, most users will likely have limited time to invest in that task. This becomes more problematic as the already large number of models available in the literature continues to increase.

A more practicable approach is to evaluate the suitability and potential performance of each model in estimating loss, for a given application setting, based on its properties. This is advantageous, as it does not require that each model be tested explicitly, and can instead be achieved by making use the information contained in a model metadata catalogue such as the ones developed by Gerl et al. (2016) or Pregnolato et al. (2015). However, models differ at various levels, and a model that is potentially superior regarding some of its properties may be inferior in terms of others (see Sect. ). Consequently, directly evaluating the potential performance of flood loss models is arduous, and currently no established procedure exists to this end. In this subsection, we address this gap by proposing a framework to rate a set of flood loss models based on their properties. The framework is described as follows:

a probability tree of model properties is set up through expert elicitation. A set of N independent properties that characterize flood loss models and that are likely to be informative for model performance are identified (e.g. damage metric). For each property (i.e. tree node) n, a set of mutually exclusive and collectively exhaustive categories are defined (e.g. relative and absolute). A subjective probability is then assigned to each category, corresponding to the degree of belief that a model that falls into that category will offer higher predictive performance than if it did in others. It follows that for each property n, the probabilities of the different categories sum to 1. Each path of the tree will have an associated probability that is obtained through the product of each node's probabilities pn along the path, therefore reflecting the degree of belief that this combination of model properties is the one that should be used;

We apply this framework to the models and test cases presented in Sect. . We first propose a probability tree referring to flood loss models for buildings. It condenses expert knowledge and current state of the art in flood vulnerability of buildings, as well as experience from previous model transfer studies. The selection of properties and categories aims to balance comprehensiveness, objectivity and simplicity. Figure 3 presents the different properties, a succinct justification of their potential relevance in assessing model performance, and the respective categories and assigned subjective probabilities. Note that the maximum partial score that can be assigned to a model for properties 1 and 2 (shown in Fig. 3) depends not only on the model but also on the hazard and exposure data sets. For example, when in a certain application case only water depth data is available, loss models that use additional explanatory variables (e.g. velocity) should not be rated higher. We then use this setup to rate the flood loss models. The results are shown in Tables 5 and 6.

While model properties are expected to be informative for performance, they are not presumed to explain it fully. However, if model properties do have usefulness in assessing the performance of models in relation to one other, some degree of correlation between model scores and different performance metrics should exist. We evaluate this using the Spearman's rank correlation coefficient rs respectively between the scores shown in Tables 5 and 6 and the error metrics shown in Tables 3 and 4. Results show a significant strong negative correlation between the variables (-0.79 < rs < -0.51, p < 0.01), which suggests that model rating based on expert judgement is indeed informative for model performance. Note that no attempt was made to maximize correlations by fine-tuning the subjective probabilities, as not only would those not correspond to the experts' degrees of belief, but more importantly, because that would be no more than an exercise in overfitting to these two case studies. This topic is revisited in Sect. .

In this exercise, the models and application cases described in Sect.  are used. For the construction of the various multi-model ensembles, we mimic the most common practical situation whereby it is necessary to estimate losses for a certain scenario for which past observational data is not available. Because in such situation, the skill of the individual models is not known, the potential suitability of each model for inclusion in a multi-model ensemble is evaluated through their properties, following the framework proposed in Sect. . Accordingly, different ensembles with increasing number of members are built, by including models sequentially from highest to lowest scores, according to Tables 5 and 6. Models with the same score are added to the ensemble simultaneously. The ensembles of different sizes constructed for each case study are shown in the x axes of Fig. 4, where 1 refers to the highest-ranked single model.

Losses given by ensemble means are estimated using two approaches: firstly, by assigning equal weights to all models, and secondly, by weighting them differently. Concerning this point, we now present some considerations. In the construction of an equal-weighted multi-model ensemble, the underlying hypothesis is that each model is independent and equally skilful, whereas this condition is most often not satisfied. For this reason, adopting different weights may increase the quality of multi-model predictions. However, finding optimal weights is not straightforward, and previous studies show that weighting models differently may result in different outcomes ranging from slight increases to degradation in performance (Doblas-Reyes et al., 2005; Hagedorn et al., 2005; Knutti et al., 2010). Here, we aim to assess how weights affect ensemble-mean performances in estimating flood loss, again by reproducing a practical situation where the skill of models in a certain future application is not known. Therefore, assigned weights instead reflect the user's confidence in each model (Marzocchi et al., 2015). Because the framework proposed in Sect.  provides scores that are proportional to relative degrees of belief among models, in principle they may be used as weights. This is achieved by normalizing the weights of the participating models in each ensemble so that they sum to 1 (Spillatura, 2014). As mentioned in Sect. , in this study we aimed to ensure model independence by selecting a set of models developed independently, by different authors, using non-overlapping datasets (Cotton et al., 2006; Palmer et al., 2004). We therefore assume that possible model dependences are not relevant and have no bearing on the weighting scheme. Section  further discusses the effect of model weighting on ensemble-based loss estimation.

Ensemble-mean performances are calculated in terms of RMSE and MBE, which are shown in Fig. 4 for the ensembles of different sizes – starting with a single model, the highest ranked for each case – and using the two weighting schemes described above. A number of observations can be made from this figure. Firstly, multi-model ensembles of any size, built by adding models with the highest degrees of belief first, considerably outperform the highest ranked single model in terms of both RMSE and MBE. This is observed for both application cases, the only exception being the MBE of some ensembles in the Caldogno case. Secondly, the performances obtained using the two different weighting approaches is mixed; while in some cases there is improvement by weighting ensemble members differently, in others the opposite is observed. The weighting approach generally does not have a significant impact on error metrics, especially when compared to the model selection. Thirdly, in both cases, the largest improvements in ensemble-mean performances are obtained after the first few highest ranked models are added. In relative terms, the impact of including additional models after that is lower. For example, in the Mulde and Caldogno case studies, the best performances are obtained with ensembles using respectively the highest-scoring four and six models. From a practical point of view, this is a particularly interesting finding because, as mentioned previously, it may not be feasible to implement a large number of models, and users may therefore be interested in parsimonious ensembles with the least number of models that lead to high predictive skill. However, in terms of probabilistic estimates of loss, smaller ensembles are less useful, which also needs to be taken into account when deciding on which ensemble size to use, as further discussed in Sect. .

Note that from here on, the equal-weighted expert-based multi-model ensembles shown in Fig. 4 will be used as a basis for other analyses and further discussion, and for the sake of brevity will be referred to as EEM-ensembles.

Some of the above observations draw comparisons between multi-model ensembles and individual models, for which the highest ranked single model is used as reference. Even though that model may not necessarily correspond to the highest performing model (which it does not in either of the application cases used here; see Tables 3, 4 and 5, 6), in a practical application case, users have no way of knowing which model is the “best”. The above results very clearly demonstrate that in such situation, using a multi-model ensemble is preferable. However, it is also insightful to assess how the constructed multi-model ensembles perform in relation to the other single models. Therefore, in Fig. 5, the error metrics of the predictions given by EEM-ensemble-means and single models are presented, showing that the former consistently outperform the latter. Note that ensembles are not expected to outperform every single model in every possible situation, and it is possible that in some application cases, certain models have such high accuracy that combining them with other models results in lower performances. The problem is that it is usually not possible to identify such models beforehand. For example, in the Mulde case, the Luino model slightly outperforms the constructed ensembles in terms of RMSE. This model consists in a simple stage-damage function that refers to a single building type, and was derived from data relative to a flood in Italy. Therefore, it is not expectable that it would consistently perform as well if applied to other analogous case studies. Overall, better performances should be obtained by using multi-model ensembles (Hagedorn et al., 2005).

The framework proposed in Sect.  and the subjective probabilities proposed in Fig. 3 provide a basis for model selection and weighting in the development of multi-model ensembles. In Sect. , we constructed various ensembles using this approach and evaluated their performance in estimating loss. However, in principle, it is possible that multi-model ensembles developed differently, i.e. by selecting different models and/or assigning different weights, would have higher skill. To investigate this issue, we simulate a so-called state of non-informativeness in terms of model suitability. This consists in assuming we have no knowledge about how particular model characteristics might affect model predictive performance (Scherbaum and Kuehn, 2011), and therefore have no way of rating models. Accordingly, we implement a probabilistic sampling procedure that, for a large number of realisations, randomly generates weights for individual models regardless of their properties. On this basis, model ensembles are built and their predictive performance is calculated for the Mulde and the Caldogno case studies. The weight generation follows the stick-breaking method, whereby models are first randomly ordered and then assigned weights sequentially. For each model, the weight is drawn from a continuous uniform distribution with a minimum value of 0 and a maximum value of 1 minus the sum of weights that have already been assigned. This approach, based on a large number of realisations, aims to cover all possible ensembles that can be constructed using the 20 flood loss models from Table 1, using not only different weighting approaches (i.e. ensemble members weighted both equally and differently) but also different combinations of models. The latter is because according to the stick-breaking method, once the model weights sum to 1, all other models receive a weight of 0 and are thus not included in the ensemble.

Scatter plots of the RMSE and MBE that result from the above procedure are presented in Fig. 6 for both case studies. The same error metrics regarding the EEM-ensembles and the single models are also included. The plots show that a wide range of possible outcomes in terms of RMSE and MBE exist when random weights are assigned to models within the framework of a state of non-informativeness. While the lower bounds of the resulting convex hull are defined by the error metrics of the lowest-performing models, the upper bounds (i.e. highest performances) are given not by any single model, but instead by multi-model ensembles, as expected. In this regard, it is clear that the model rating framework based on expert judgement and subjective probabilities proposed in Sect.  add value to the ensemble development process. Indeed, ensembles that are constructed by adding models prioritized in terms of potential suitability (shown in Fig. 4) are among the highest performing ensembles, considering all the existing possibilities. It is interesting to highlight that the simple unweighted mean of all models also performs relatively well, which suggests that if no knowledge is available on model properties and/or on how they influence performance, it is better to include all models than to wrongly select them.

The plots also show that it is possible to create certain ensembles that lead to better skill in relation to the ones developed based on expert judgement. However, the potential relative degree of improvement is very low in both test cases, more markedly so in the Caldogno case, which reinforces the idea that the approach proposed in Sect.  provides a good basis for ensemble construction. We do not attempt to maximize the performance of the constructed multi-model ensembles based on the results obtained in this exercise, as this would be of little relevance. Analogously with the “best” model discussion in Sect. , in a practical application the ensembles cannot be tested beforehand. Finding specific weights that maximize performance for the Mulde and the Caldogno case studies would consist in pointless overfitting, as such weights necessarily vary from case to case. In addition, it is likely that such weights would not make sense from the perspective of an expert. Instead, the objective here is that ensembles are constructed in a manner that leads to good performances in all situations, which the results support. Finally, Fig. 6 corroborates that correctly selecting models for an ensemble is more important than weighting them. The EEM-ensembles, which result from model selection only, display error metrics close to the minimum obtainable from a wide range of possible outcomes. In comparison, further improvements that could possibly be achieved by assigning different weights to ensemble members are very small.

Regardless of the method that is used to obtain probabilistic estimates from multi-model ensembles, it is first important to evaluate the “raw” ensembles, with minimum interference from the ensemble interpretation model that is used. This can be achieved using the continuous ranked probability score (CRPS) (Bröcker, 2012; Hersbach, 2000). We calculate the CRPS for the EEM-ensembles, and present the results in Fig. 7. This is done for the Caldogno case study, as the low number of data points in the Mulde case (19) are insufficient for such analysis.

The probabilistic skill of the ensembles is observed to have an increasing trend (i.e. decreasing CRPS) with the number of participating members. This is to some extent expected, as ensemble size is known to have an effect on probabilistic skill scores, which is explained by the fact that probabilistic estimates derived from ensembles become more unreliable as the size of the ensemble gets smaller (Weigel, 2012). This highlights the need of using a considerable number of models when the objective is to obtain reliable (i.e. statistically consistent) probabilistic estimates of flood loss. Another requirement to achieve this is that the ensemble itself is reliable, in the sense that ensemble members and observations are sampled from the same underlying probability distributions or, in other words, that they are statistically indistinguishable from each other (Leutbecher and Palmer, 2008). Even an ensemble of infinite size is unable to yield reliable probabilistic estimates if its members are not reliable (e.g. if they are heavily biased). For illustration, we assess reliability considering an ensemble comprising all 20 models implemented in this study using the rank histogram, which is shown in Fig. 8. As expected, the ensemble is not perfectly reliable; however, the counts do tend to oscillate around nm+1=29621, which suggests a reasonable degree of reliability. In addition, the ensemble appears to be slightly over-dispersive, due to an overpopulation of central ranks of the histogram.

Finally, we illustrate the simplest approach to obtain a probabilistic distribution of flood losses using a multi-model ensemble. For each building, a value of loss is randomly generated using the reverse transform sampling method, whereby a number u∼[0,1] is sampled from the standard uniform distribution, and the corresponding quantile is sampled from the empirical cumulative distribution function (ECDF) of losses given by ensemble members through linear interpolation. The losses for each building are then summed up, and a total loss is obtained. This process is repeated a large number of times, yielding a loss distribution for the flood event. The results for the Caldogno application, based on 10 000 realisations, are shown in Fig. 9 in the form of a histogram and ECDF of total loss.

Statistical post-processing techniques may be used to improve the reliability of probabilistic predictions. This is common practice in the field of numerical weather prediction, for example. However, in that case, relatively long time series of past observational data for a certain variable (e.g. temperature) at a certain location are usually available, and such data continue to be collected, which allows the predictive system to be calibrated and the forecasts verified. This is in contrast with the case of flood loss estimations, where loss models necessarily need to be transferred due to the rarity of the events and the difficulty in obtaining data. In the particular case of probabilistic loss estimates based on ensembles, it is therefore necessary to investigate how best to improve their reliability for future applications by considering data from previous flood events often occurring in different contexts. In addition, as mentioned previously, the reliability of probabilistic estimates may also be improved by using a more sophisticated ensemble interpretation method (i.e. kernel dressing or parametric distribution fitting). However, the most appropriate approach to do this in the case of flood loss modelling also needs to be investigated. These topics are beyond the scope of this article.