Preprints
https://doi.org/10.5194/nhess-2021-215
https://doi.org/10.5194/nhess-2021-215

  16 Aug 2021

16 Aug 2021

Review status: this preprint is currently under review for the journal NHESS.

Methodological and conceptual challenges in rare and severe event forecast-verification

Philip Alexander Ebert1, and Peter Milne1, Philip Alexander Ebert and Peter Milne
  • 1Division of Law and Philosophy, University of Stirling, UK
  • These authors contributed equally to this work.

Abstract. There are distinctive methodological and conceptual challenges in rare and severe event (RSE) forecast-verification, that is, in the assessment of the quality of forecasts involving natural hazards such as avalanches or tornadoes. While some of these challenges have been discussed since the inception of the discipline in the 1880s, there is no consensus about how to assess RSE forecasts. This article offers a comprehensive and critical overview of the many different measures used to capture the quality of an RSE forecast and argues that there is only one proper skill score for RSE forecast-verification. We do so by first focusing on the relationship between accuracy and skill and show why skill is more important than accuracy in the case of RSE forecast-verification. Subsequently, we motivate three adequacy constraints for a proper measure of skill in RSE forecasting. We argue that the Peirce Skill Score is the only score that meets all three adequacy constraints. We then show how our theoretical investigation has important practical implications for avalanche forecasting by discussing a recent study in avalanche forecast-verification using the nearest neighbour method. Lastly, we raise what we call the “scope challenge" that affects all forms of RSE forecasting and highlight how and why the proper skill measure is important not only for local binary RSE forecasts but also for the assessment of different diagnostic tests widely used in avalanche risk management and related operations. Finally, our discussion is also of relevance to the thriving research project of designing methods to assess the quality of regional multi-categorical avalanche forecasts.

Philip Alexander Ebert and Peter Milne

Status: open (until 01 Nov 2021)

Comment types: AC – author | RC – referee | CC – community | EC – editor | CEC – chief editor | : Report abuse
  • RC1: 'Comment on nhess-2021-215', Krister Kristensen, 14 Sep 2021 reply

Philip Alexander Ebert and Peter Milne

Philip Alexander Ebert and Peter Milne

Viewed

Total article views: 287 (including HTML, PDF, and XML)
HTML PDF XML Total BibTeX EndNote
233 46 8 287 1 1
  • HTML: 233
  • PDF: 46
  • XML: 8
  • Total: 287
  • BibTeX: 1
  • EndNote: 1
Views and downloads (calculated since 16 Aug 2021)
Cumulative views and downloads (calculated since 16 Aug 2021)

Viewed (geographical distribution)

Total article views: 259 (including HTML, PDF, and XML) Thereof 259 with geography defined and 0 with unknown origin.
Country # Views %
  • 1
1
 
 
 
 
Latest update: 16 Sep 2021
Download
Short summary
There is no consensus about how to assess the quality of binary (yes/no) rare and severe event forecasts, i.e. forecasts involving natural hazards like tornadoes or avalanches. We offer a comprehensive overview of the challenges we face when making such an assessment and provide a critical review of existing solutions. We argue against all but one existing solutions to assess the quality of such forecasts and present practical consequences to improve forecasting services.
Altmetrics