Comment on nhess-2021-237

The paper “Effective uncertainty visualization for aftershock forecast maps” presents an empirical study of how different visualization styles affect interpretation of the uncertainty associated with aftershock forecast maps. Three uncertainty visualisations are tested through an online survey that was designed to address stakeholder communication needs for “sure bet” and “surprise” aftershock events. The task performance metrics reveal that presenting lower and upper bounds of uncertainty was more effective at communicating “surprise” events, where there is high uncertainty in the forecast.

https://doi.org/10.5194/nhess-2021-237-RC2, 2021 The paper "Effective uncertainty visualization for aftershock forecast maps" presents an empirical study of how different visualization styles affect interpretation of the uncertainty associated with aftershock forecast maps. Three uncertainty visualisations are tested through an online survey that was designed to address stakeholder communication needs for "sure bet" and "surprise" aftershock events. The task performance metrics reveal that presenting lower and upper bounds of uncertainty was more effective at communicating "surprise" events, where there is high uncertainty in the forecast.
Overall, the paper is very well written and addresses a timely and important topic in risk communication. The study design and data analysis are strong and the relevancy of the study is discussed in relation to the wider literature. It is excellent to see that the study design was informed by stakeholder needs, and the framing of exploring "sure bets" and "surprises" is novel and useful. I recommend this paper for further publication based on minor revisions that may help clarify some of the study design and its implications and limitations, as well as potentially improve digestibility of tricky uncertainty concepts.
Line 40-41: There are many previous experiments that consider user communication needs. I agree it is a strength of this study that interviews were carried out in order to explicitly inform the research design. However, that isn't immediately clear from this sentence. I would be more explicit and say that interviews were carried out with stakeholders to understand uncertainty visualisation needs so that the results can be used to inform evidence-based practice, or something along those lines. Line 95: the "skewed distribution" of aftershock rates is mentioned several times throughout the paper. I think it would be helpful to show an example plot of this distribution and it's associated uncertainty bounds. Uncertainty, especially when discussing probability distributions can be a bit of a tricky concept to digest -I think showing a graphic explanation of the "skewed" distribution, it's upper and lower bounds, and visually highlighting areas where the "sure bet" and "surprise" events lie could help clarify the terms.
Line 138: Thompson et al. (2015) explore user interpretation of hazard curve graphics with upper and lower bounds alongside probability maps for ashfall forecast maps, though they don't explore the concept in depth or geospatially.
Line 195: It may be worth mentioning that the higher uncertainty could also potentially result in fewer aftershock forecasts than forecasted?
Line 318, Fig 2: It would be helpful to break Fig 2 into part A and B, so that you can refer to the "rate only" map specifically, otherwise at first read it appears you are only showing 3 of the 4 visualisation conditions tested Line 325: The video tutorial explanation is a great approach to help ensure participants are fully informed before participating in the study Line 452, Figure 6: I find this figure very difficult to interpret…Is there a way to simplify? The scale "% Judgement in favour of…" is not immediately clear…Is this between the two locations? Could consider using a graph design that enables you to show both locations for each trial (e.g., high and low rate; high and low uncertainty) for the reader to compare? It may be helpful throughout the figures to say "aftershock rate" instead of just "rate" for clarity.
Line 545: Here the authors state that readers were less accurate reading uncertainty than other visualisations, but the abstract says at line 47 that all visualisations performed equally in communicating uncertainty…maybe double check consistency / language here, as I understand "sure bet" to still be communication of uncertainty Line 545 forward: Throughout the discussion the authors speculate on reasons for potential miscommunication, it would be worth noting in section 4.5 on limitations and future research that this study did not collect any open ended qualitative feedback, and that this may provide insight into some of these visual communication challenges.
Line 629-630: "Futhermore…" Can you clarify this sentence? Evenly binned in regards to what?
Line 620: Can you discuss or note to what type of situation this might apply in practice?
Line 672-673: The conclusion starts by stating that there are implications for nontechnical users, then the following sentence outlines implications for technical users. Consider reframing/organising for clarity.
Line 680: The concluding paragraph talks about "intervals" and "intervals-based" design, but the results and discussion talk about "bounds", consistency would improve clarity of these statements Lastly, the results about the bounds/interval map are very interesting but there is limited discussion of how these might inform design in practice (e.g., what are the possible implications of using two separate boundary maps labelled optimistic and pessimistic in a high stakes crisis communication environment? Could these be misinterpreted or separated?)