Thanks a lot, Thorsten Wagener, for your thoughtful comments. We respond to your comments as follows:

[1] Our understanding of validation is explained in lines 53 – 71 (“… evaluating its [the model’s] ability to achieve its intended purpose. In essence, the evaluation of an FHRA model's validity is determined by its fitness for its intended purpose, reframing the criteria for assessment from correspondence to reality to alignment with decision-making needs …” and “… Our focus on a fit-for-purpose approach follows earlier arguments … the process of structured reasoning about the level of confidence needed to support a particular decision and the credibility of the assessment of risk in that context …”. However, we agree that a more explicit definition of the term validation is helpful for the reader, and we will re-write these 2 paragraphs accordingly. For a more explicit definition, we will follow Eker et al. (2018) that you propose in your comment [9].

Actually, we don’t use the term verification in our paper, but verifiability which is explained in Table 2 and Figure 1.

[2] On the question whether we should use the term validation: Yes, Oreskes et al. argue that the terms validate and verify are problematic: “ … both verify and validate are affirmative terms: They encourage the modeler to claim a positive result . And in many cases, a positive result is presupposed. For example, the first step of validation has been defined by one group of scientists as developing "a strategy for demonstrating [regulatory] compliance". Such affirmative language is a roadblock to further scrutiny. A neutral language is needed for the evaluation of model performance …” While Oreskes et al. note this problem, they don’t really suggest to substitute validation by evaluation. Evaluation is rather a procedure. Oreskes et al. (1994) mention that the term validation implies legitimacy, e.g., a contract that has not been nullified is a valid one. They mention, however, that this term is often misused by using it in a sense of verification, i.e., consistency with observations, and in a sense that a valid model is a realistic representation of physical reality, i.e., “truth”. Our definition of validation as being fit for purpose resonates with the notation by Oreskes et al. in a sense that we agree on a “contract” for model use for a specific decision-making purpose, given certain model properties and quality ensuring procedures, and this makes it legitimate or valid.  In addition, the term validation is extremely widespread. The majority of papers discussing the simulation of environmental systems use validation, including those papers that agree that “do not see how there could be a single way to simplify the system reach a “true” model”. Thus, to better connect to the existing literature and terms used, we prefer to use validation instead of the evaluation. However, we will add a disclaimer that validation has this affirmative touch.

On the discussion of “true”: We have checked how we use “true” or “truth” in the manuscript. There are only 3 instances where we use them:

“ … Firstly, validation can establish legitimacy but not truth. Truth is unattainable because …” (Line 150) and “… The ideal model-building process utilizes an initial model to make testable predictions, then takes measurements to test and improve it (Ewing et al., 1999). This predictive validation approach appeals because the modeler is unaware of the truth at the time of the model experiment and is therefore not subject to hindsight bias…” (Line 190).

We will keep the first 2 instances, as we basically say what you mean (“A model cannot be true”). Concerning the third instance, we will substitute “truth” by “measurements”.

[3] We will delete this sentence in the revision as we don’t need it to make our point, which is that validation can establish legitimacy but not truth.

[4] We agree that there is a difference between subjective choices and a subjective process and will reformulate as follows: “Validation therefore includes subjective choices”.

[5] It is indeed an important question how we decide on appropriate thresholds. Here, a basic problem is that the specific thresholds and the ways how to decide on them certainly vary between different contexts. We believe that our framework can help to decide, in a certain context, whether the specific model is valid enough. We follow Howard (2007) who discussed the related problem of what a good decision is. According to Howard, a decision should not be strictly judged by its outcome: A good decision does not always lead to a good outcome and a bad decision does not always lead to a bad outcome. Instead, a good decision is governed by the process that one uses to arrive at a course of action. Howard then defined 6 (decision quality) elements and argued that good decisions are those in which all of these elements are strong. Similarly, we think that our framework can support the discussion about the degree of validation of a model. We think that a good validation is governed by the process that one uses to evaluate a model. Applying our framework (Table 2), we argue that a specific model is validated when all 7 criteria are fulfilled to an extent that is appropriate in the specific context. Of course, the problem still remains what “appropriate in the specific context” means. However, given the large range of contexts where FHRAs are performed, we think that it is not possible to discuss the many ways that parties involved in a FHRA might decide whether a model is valid enough.

In the revised version, we will add a discussion on your important question along the lines outlined above.

[6] Yes, at some point in the process of validating a FHRA, there is typically a decision that the specific model is valid enough (despite its imperfect suitability). And such decisions are required, because a flood protection measure needs to be designed (or any other real-world decision needs to be taken) based on a concrete model output (which can be a single number, a probability distribution, a range of what-if scenarios, etc.). In such a situation, one could consider the degree to which model is valid (its validity) in the decision context. Often, this is already done. For instance, reliability engineering (e.g. Tung, 2011) considers (aleatory and epistemic) uncertainty in the design of structures. In a situation where the available model is less able to reproduce the observations, one can consider this lower validity in a wider probability distribution of the load (external forces or demands) on the system or the resistance (strength, capacity, or supply) of the system. This, in turn, will lead to higher design values due to our high uncertainty represented in the specific model. In a situation where one has several, alternative models, each associated with a measure of how valid they are (e.g. by quantifying their agreement with observations), one can weigh these models to obtain the concrete model output required for a specific decision. We agree with Thorsten Wagener that the full acceptance of models is a key problem and we will extend our manuscript by a discussion on that problem, including some reflections on how one could incorporate the degree of validity in decision-making contexts.    

We think that stating this range (by specifying the range of return periods, failure mechanisms etc., for which data exist, should be specified, as should those cases for which observations are unavailable) is extremely helpful. However, we will clarify in the revised version that we don’t assume that the model is correct even when it is valid enough to be used for decisions in a specific context.

[7] We have mentioned the challenge of the lack of data at several locations in the original manuscript, most prominently in Lines 92 – 100: “… One fundamental problem is that flood risk, i.e. the probability distribution of damage, is not directly or fully observable. Extreme events that lead to damage are rare, and the relevant events may even be unrepeatable, such as the failure of a dam (Hall and Anderson, 2002). The rarity of extreme events results in a situation characterized by both limited data availability and increased data uncertainty. This uncertainty relates to data against which the flood model can be compared. For instance, streamflow gauges often fail during large floods, and losses are not systematically documented and reported losses are highly uncertain. In addition, input data is often insufficient for developing a viable flood model. For example, levee failures depend on highly heterogeneous soil properties, and levee-internal characteristics are typically unknown. Thus Molinari et al. (2019) conclude that “a paucity of observational data is the main constraint to model validation, so that reliability of flood risk models can hardly be assessed…”. Our framework is our answer to this challenge. The framework goes beyond the current, most prominent view on model validation (which is strongly focussed on data validation, i.e. comparing simulation against observation) by adding validation elements and criteria (see Figure 1) that can be applied without observations.

In the revised version, we will discuss more clearly how our framework addresses the question of validating without data.

[8] Thanks a lot for this wider perspective on sensitivity analysis which we will consider in the revised version.

[9] and [10] Thanks a lot for these references, which is highly relevant and will be included in our revision.

References:

Howard RA (2007) The foundations of decision analysis revisited. In: Edwards W, Miles RF, Von Winterfeldt D (eds) Advances in decision analysis: from foundations to applications. Cambridge University Press, Boston, pp 32–56

Tung, Y.-K. (2011). Uncertainty and reliability analysis in water resources engineering. Journal of Contemporary Water Research and Education, 103(1), 4.

Thanks for these helpful comments. We respond to these comments as follows:

[1] Our paper is not a review on the validation of flood hazard and risk assessments, but a perspective paper which attempts to offer a broader view on validation. To this end, we have tried to cite examples that cover the entire range of flood hazard and risk assessments. However, as validating risk is more difficult than validating hazard, we agree that putting more emphasis on examples that deal with validating the modelling of flood consequences will make the paper more useful. In the revision, we will thus add more examples on validating consequences / risk.

[2] The point that engineers tend to exaggerate flood risk in order to benefit from large risk numbers is highly interesting. However, we could not find evidence for the statement of the reviewer “… Indeed, it seems that a review of most models show widespread exaggerations over anticipated consequences …”. There are very few papers that point in this direction. (One example is ‘Penning-Rowsell, E. C. (2021). Comparing the scale of modelled and recorded current flood risk: Results from England. Jour. Flood Risk Manag’. It compares the modelled numbers of flood risk for England with loss figures quantified in terms of insurance claims data and finds that modelled results are between 2.06 and over 9.0 times the comparable flood losses measured in terms of the compensation paid.)

While we agree that the danger of exaggeration exists, ethical standards of the engineering profession and the public and political scrutiny of large infrastructure projects serve as checks against such behaviour. In many regions/countries, there is a strong emphasis on demonstrating the effectiveness and cost-benefit ratio of flood protection measures, which suggests a focus on justifying any proposed measures rather than exaggerating risks or promoting unnecessary construction. However, we will extend the manuscript and add a short discussion on this point that the reviewer raised.

[3] We don’t understand the statement: “… Indeed, the final paragraph of that section rather implies that modelling extreme events in real time is not likely to be credible…”. The final paragraph is “… The use case of emergency flood management exemplified in Table 3 reflects the situation that flood models for the management of extraordinary situations cannot rely on typical elements of validation, such as comparing model simulations with observed data. Only rarely does observed data about inundation, defence failures, and impacts exist for a particular region. This is precisely why it is all the more important to safeguard the usability and credibility of the models applied…” Why does this paragraph imply that modelling in real time is not credible? We think that models can be useful and credible even when we don’t have (many) observations.   

We have chosen the example of emergency management deliberately because this is an area without much data to validate models. We think that our framework is helpful exactly in such situations, when there is little data and when the typical approach of validation (compare against observations) is not possible. However, we will extend this example by going more into depth when presenting the validation elements.