Overall this is more convincing and, by and large, more balanced than the previous version. The referees both have some concerns, but they can be dealt with fairly straightforwardly. For my own part, I think Section 2 still needs a bit of work in order to present a coherent message that is relatively uncontentious. I'll come to this below. First, the referee comments.

Referee 1:
==========

I confess that I asked Mike Poole to look at this because I knew that (a) he has his head screwed on (b) he would offer a very complementary viewpoint to that expressed in the paper. He has not disappointed! His report has revealed (as I hoped it would) some opportunities for the discussion to be tightened up. Having considered his comments and the authors' responses to them, I suggest the following course of action:

- Include some examples of visualisations, complementing those in the companion paper (e.g. chosen to represent generic examples of what you consider to be good practice).

- "Epistemic and aleatory uncertainties". The referee's main point here is already acknowledged in the paper (e.g. para beginning on L15 of P3), and I'm satisfied with the authors' response - in particular, their point that "even if the resulting uncertainties are wide, they are not necessary [sic] unhelpful". But the referee makes a very important point - which doesn't come out from the paper in its current form - that epistemic uncertainties are *potentially* reducible [whereas aleatory uncertainties are not] (emphasis added, along with amplification). Given that many applied scientists don't understand what uncertainty even means, it would be helpful to make this point somewhere in the paper: thus, in terms of "best practice", there's an opportunity to consider some fundamental principles of experimental design (in terms of the collection of observations, or the configurations of model runs) that aim specifically to maximise the precision of the resulting inference while minimising bias.

- "Probabilistic vs non-probabilistic methods". The referee raises a very interesting issue here. I broadly agree that "probability is mathematically the correct way to represent uncertainty": at least, I haven't seen any alternatives that offer more in an applied setting. In my view, some of the alternatives (such as fuzzy logic) are just less ambitious and, actually, less engaged with the realities of practical decision-making. For example, as far as I'm aware there's no rigorous formal framework for decision-making based on fuzzy logic, whereas there is such a framework based on probability.

A related point is that as a "mainstream" technique, probabilistic reasoning has been subjected to much more stringent critical scrutiny than some of the alternative approaches that have been suggested. E.g. I'm prepared to stick my neck out here and claim that (e.g.) Dempster-Schafer theory will always be a bit of a niche subject: it's not going to attract a barrage of criticism therefore, simply because only a tiny minority of practitioners are ever going to know about it.

I'm not entirely convinced by the authors' response to this comment from the referee. The response "We can envisage applications where the use of bounded fuzzy possibilities would lead to more constrained and realistic uncertainty estimates than unbounded probabilities (that can in the worse cases give probability to negative values of variables that can only be positive)" is at best disingenuous. The "worst case" scenario here could only be perpetrated by an analyst who is not qualified to carry out this type of work. Unfortunately, it's not unknown - but there is no place for even an implicit endorsement of this in a paper with "What should constitute good practice?" in the title.

Having said that: on the basis of my personal knowledge of everyone involved, I don't think there's really too much disagreement here. I think the issue could be resolved just by acknowledging some of the limitations of the alternatives to probability (e.g. that they're less ambitious), as outlined above.

- "Expert elicitation": the referee indicates three things that are an essential part of an expert elicitation process. It would be worth incorporating something in the paper to recognise these points. The authors' response suggest that they're prepared to do this: I would encourage them to do so. I also strongly agree with the referee's suggestion to include a paragraph on cognitive biases. I can provide a copy of his reference [1] if this would be helpful.

- "Assumptions": the issue here is one of vocabulary, I think. The referee is using the word in a more restrictive sense than the authors. I found this quite interesting - I'm used to differences in interpretation of many words (e.g. "parameter", "prediction"), but this is a new one for me! But if the referee can read the word in such a way, so can other potential readers of the paper. The way to deal with this is to be absolutely explicit, early on, about the scope of your use of the word. You don't need to clarify every single use of the word (as you suggest in your response): just define your terms at the outset.

Referee 2:
==========

Point 1: the referee finds the paper hard to read in places. The authors' response is along the lines that this is inevitable given the subject matter. However, this referee is very smart and very experienced: if (s)he finds the discussion dense then many other readers will have a similar reaction, and I think the comment needs to be taken seriously. The suggestion to define key terms when used for the first time is a good one: there's no need to avoid the use of terms like "simulator", just make sure they're defined.

Point 2: I sympathise with the referee here about the lack of balance in the paper. However, it's stated at the outset that the paper is an opinion piece representing the views of the authors so I think this is probably OK.

Point 3: nothing further to add - although some of the responses to comments from Referee 1 might help to address this issue.

Point 4: it looks to me as though the referee may have misunderstood the authors' intention in the last line of page 10, and the authors haven't realised the misunderstanding. I read this as saying that you might have a bunch of GCM projections and decide to give them all equal weight, or you might have a bunch of downscaled projections and give *them* equal weight. I think the referee has read it as saying that you could have, say, a rainfall projection from a GCM and then another projection from and RCM (and possibly yet another from a stochastic weather generator) and give *these* different weights. Of course, this is indeed a possibility, but it would be a bit odd. To remove the confusion, you could change "(from different Global Circulation Models, or downscaling procedures, or ways of implementing future change factors)" to "(e.g. from different Global Circulation Models, or from different downscaling procedures, or from different ways of implementing future change factors)".

However, the referee makes another important point in this comment: the need to think about "what the models are supposed to represent". This is an issue that isn't really highlighted in the paper at present: it is, I suppose, implicit in the idea of using expert elicitation, but there is also a fundamental responsibility on the analyst to understand what a particular model is designed to do, to understand its limitations and to ensure that its use is justified in the situation at hand. This might sound obvious. Unfortunately however, there are plenty of examples in the literature of techniques being misapplied, or of extensive intercomparison exercises between techniques that are in many cases inappropriate because they were designed for a different problem (the downscaling community is rife with this kind of thing). The authors may therefore care to consider a "recommendation" that epistemic uncertainties can sometimes be reduced simply by considering what specific models were designed to do.

Point 5: nothing further to add. The authors propose to modify the text with appropriate references - this is fine.


Additional comments:
====================

Finally, my own additional comments on Section 2. I have two main concerns with it in its current form, as follows:

- The six steps at the bottom of page 2 seem fairly uncontroversial. However, in a paper with "What should constitute best practice?" in the title it would perhaps be helpful to ensure that each of the corresponding subsections contain clear recommendations. For example, there are no recommendations at present in Section 2.2, and the recommendations in some of the other subsections would benefit from being flagged more clearly.

- Early in the section, there are still some fairly strong statements that implicitly criticise specific methodologies, when in fact the issue is about the inappropriate / unskilled use of those methodologies. This is acknowledged later on in Section 2.5 (specifically, from lines 30 onwards of page 9), but it seems a bit out of place here - Section 2.5 is about assessing the robustness of decisions to the chosen assumptions, which is a completely different question from the need to use analysis methods that account appropriately for the salient features of the problem (including whatever is known about the potential limitations of the models and data). In my view, the material from P9L30 to P10L16 would fit more naturally in Section 2.4 - it is really about the workflows and choice of methodology. By the time we get to line 17 of page 10 - talking about the potential for future surprise - we are indeed back to the "robustness" issue, and I have no problem with the subsequent discussion of potential techniques that might be used to address this.

I refer above to "fairly strong statements" in parts of Section 2.2: as well as moving some of the material from Section 2.5 into Section 2.5, some work is needed to make these statements more precise / balanced. Specifically:

- P3 lines 29-30: "the assessment, and the analyst, can always
be defended ..." and the subsequent "Effectively, the analyst
cannot be wrong" makes it sound as though the "capricious
nature of reality" defence is valid. It isn't, of course -
it's lazy and complacent. The truth is that the observation
of events, or combinations of events, to which we have
previously ascribed very low probability will invariably lead
a prudent analyst to consider carefully whether these events
are really very unlikely or whether there are substantive
deficiencies in the process by which the probabilities were
calculated. It might be worth including a comment to clarify
this.

- P4, line 12: please define precisely what you mean by
"disinformative", and clarify that the problem highlighted
here is not with the model or the data: it is with the
analyst who fails to acknowledge their imperfections. I asked
for this last time around! The point is that *uncritical*
use of imperfect models and data is problematic. Given that
all models and most data are imperfect, I would say that
there is a responsibility on the analyst to be explicit about
those imperfections and, in an ideal world, to account for
them appropriately in the analysis. I'm not saying it's easy,
but it's certainly possible in the examples cited further
down the page. E.g. in line 21 you say that "including those
events in conditioning the simulator would lead to incorrect
inference". This is only true if you do the inference naively.
I must insist that this point is acknowledged: indeed, I'd
say that this should be one of the recommendations for the
section (see my first main concern above). For the
rainfall-runoff example of course, omitting the offending
events could be considered as one way to "acknowledge" the
deficiencies in the observations. In this situation however,
*without further information* (e.g. some knowledge of the
measurement process in relation to features of these specific
events) I would personally be worried about the quality of
the observations for the remaining events and I wouldn't trust
*any* model calibration based on those data. The fact that
the observations are not physically incompatible does not in
itself indicate that they're fit for purpose!

- P4, line 29: "Normal statistical measures ... may not be
appropriate ...". Entropy measures don't tell us anything
about information content in the sense that is being discussed
here, which is the extent to which the data constrain our
understanding of the system (Fisher or Godambe information
are much more useful concepts in this respect - for Godambe
see, for example, my contribution to Chapter 4 of the Beven &
Hall book on uncertainty in flood risk management). But in any
case, most of these measures are fundamentally defined in terms
of the assumed joint probability density function of the
observations: the "normal" versions to which the authors refer
are just designed for use in very simple situations. If the
errors are complex, nonstationary etc. then the onus is on the
analyst to recognise this and to figure out from first
principles what the entropy / Fisher / Godambe information
should be. Once again therefore, the problem isn't with the
measures per se, it's with their uncritical use. This needs
to be acknowledged.

- P7, lines 17-19, comment on the use of "formal statistical
likelihood measures": this is the same thing again. It is not
good practice to use formal measures based on inappropriate
assumptions (e.g. for which the likelihood is computed on the
basis of a mis-specified probability model that fails to
account for the structure of the observations). Moreover, it
may be impractical to formulate a statistical likelihood
explicitly in any given situation. But it's perfectly possible
to produce formal comparisons using techniques that acknowledge
that the models under consideration are "wrong but useful": see,
for, example, DOI 10.1098/rsfs.2011.0057 (sorry about the self-
citation but this paper was written in 2011 precisely to raise
awareness of this kind of technique). The comment about "overly
simple statistical analysis" in line 23 is OK so far as it
goes, but it isn't really enough to dispel the impression that
there's somehow a flaw in *all* of the statistical ideas that
have been mentioned so far.

- P9, line 25: this statement about the Gumbel distribution isn't
strictly correct: there are examples of dependent sequences in
which the maxima are Gumbel distributed in the limit. In any
case, the Gumbel distribution only applies to a subset of
situations and nobody should be restricting extreme value
analyses to the Gumbel case these days. Suggest removing the
parenthetical remark - or replace it with "(such as the
generalised extreme value family)" without further elaboration
that may get you into trouble.

I think we've got as far as we're going to get with this now, and we'll have to accept that we're not going to agree on some of the points. I'm accepting the paper without further revision, therefore.