<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE article PUBLIC "-//NLM//DTD Journal Publishing with OASIS Tables v3.0 20080202//EN" "https://jats.nlm.nih.gov/nlm-dtd/publishing/3.0/journalpub-oasis3.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:oasis="http://docs.oasis-open.org/ns/oasis-exchange/table" xml:lang="en" dtd-version="3.0" article-type="research-article">
  <front>
    <journal-meta><journal-id journal-id-type="publisher">NHESS</journal-id><journal-title-group>
    <journal-title>Natural Hazards and Earth System Sciences</journal-title>
    <abbrev-journal-title abbrev-type="publisher">NHESS</abbrev-journal-title><abbrev-journal-title abbrev-type="nlm-ta">Nat. Hazards Earth Syst. Sci.</abbrev-journal-title>
  </journal-title-group><issn pub-type="epub">1684-9981</issn><publisher>
    <publisher-name>Copernicus Publications</publisher-name>
    <publisher-loc>Göttingen, Germany</publisher-loc>
  </publisher></journal-meta>
    <article-meta>
      <article-id pub-id-type="doi">10.5194/nhess-23-2895-2023</article-id><title-group><article-title>Avalanche size estimation and avalanche outline determination by experts: reliability and implications for practice</article-title><alt-title>Avalanche size estimation and avalanche outline determination</alt-title>
      </title-group><?xmltex \runningtitle{Avalanche size estimation and avalanche outline determination}?><?xmltex \runningauthor{E. D. Hafner et al.}?>
      <contrib-group>
        <contrib contrib-type="author" corresp="yes" rid="aff1 aff2 aff3">
          <name><surname>Hafner</surname><given-names>Elisabeth D.</given-names></name>
          <email>elisabeth.hafner@slf.ch</email>
        <ext-link>https://orcid.org/0000-0002-9554-7900</ext-link></contrib>
        <contrib contrib-type="author" corresp="no" rid="aff1">
          <name><surname>Techel</surname><given-names>Frank</given-names></name>
          
        <ext-link>https://orcid.org/0000-0001-5686-6127</ext-link></contrib>
        <contrib contrib-type="author" corresp="no" rid="aff3">
          <name><surname>Daudt</surname><given-names>Rodrigo Caye</given-names></name>
          
        </contrib>
        <contrib contrib-type="author" corresp="no" rid="aff3 aff4">
          <name><surname>Wegner</surname><given-names>Jan Dirk</given-names></name>
          
        </contrib>
        <contrib contrib-type="author" corresp="no" rid="aff3">
          <name><surname>Schindler</surname><given-names>Konrad</given-names></name>
          
        </contrib>
        <contrib contrib-type="author" corresp="no" rid="aff1 aff2">
          <name><surname>Bühler</surname><given-names>Yves</given-names></name>
          
        <ext-link>https://orcid.org/0000-0002-0815-2717</ext-link></contrib>
        <aff id="aff1"><label>1</label><institution>WSL Institute for Snow and Avalanche Research SLF, Davos Dorf, 7260, Switzerland</institution>
        </aff>
        <aff id="aff2"><label>2</label><institution>Climate Change, Extremes, and Natural Hazards in Alpine Regions Research Center CERC, Davos Dorf, 7260, Switzerland</institution>
        </aff>
        <aff id="aff3"><label>3</label><institution>EcoVision Lab, Photogrammetry and Remote Sensing, ETH Zurich, Zurich, 8093, Switzerland</institution>
        </aff>
        <aff id="aff4"><label>4</label><institution>Institute for Computational Science, University of Zurich, Zurich, 8057, Switzerland</institution>
        </aff>
      </contrib-group>
      <author-notes><corresp id="corr1">Elisabeth D. Hafner (elisabeth.hafner@slf.ch)</corresp></author-notes><pub-date><day>30</day><month>August</month><year>2023</year></pub-date>
      
      <volume>23</volume>
      <issue>8</issue>
      <fpage>2895</fpage><lpage>2914</lpage>
      <history>
        <date date-type="received"><day>27</day><month>March</month><year>2023</year></date>
           <date date-type="rev-request"><day>11</day><month>May</month><year>2023</year></date>
           <date date-type="rev-recd"><day>24</day><month>July</month><year>2023</year></date>
           <date date-type="accepted"><day>25</day><month>July</month><year>2023</year></date>
      </history>
      <permissions>
        <copyright-statement>Copyright: © 2023 Elisabeth D. Hafner et al.</copyright-statement>
        <copyright-year>2023</copyright-year>
      <license license-type="open-access"><license-p>This work is licensed under the Creative Commons Attribution 4.0 International License. To view a copy of this licence, visit <ext-link ext-link-type="uri" xlink:href="https://creativecommons.org/licenses/by/4.0/">https://creativecommons.org/licenses/by/4.0/</ext-link></license-p></license></permissions><self-uri xlink:href="https://nhess.copernicus.org/articles/23/2895/2023/nhess-23-2895-2023.html">This article is available from https://nhess.copernicus.org/articles/23/2895/2023/nhess-23-2895-2023.html</self-uri><self-uri xlink:href="https://nhess.copernicus.org/articles/23/2895/2023/nhess-23-2895-2023.pdf">The full text article is available as a PDF file from https://nhess.copernicus.org/articles/23/2895/2023/nhess-23-2895-2023.pdf</self-uri>
      <abstract><title>Abstract</title>

      <p id="d1e147">Consistent estimates of avalanche size are crucial for communicating not only among avalanche practitioners but also between avalanche forecasters and the public, for instance in public avalanche forecasts. Moreover, applications such as risk management and numerical avalanche simulations rely on accurately mapped outlines of past avalanche events. Since there is not a widely applicable and objective way to measure avalanche size or to determine the outlines of an avalanche, we need to rely on human estimations. Therefore, knowing about the reliability of avalanche size estimates and avalanche outlines is essential as errors will impact applications relying on this kind of data. In the first of three user studies, we investigate the reliability in avalanche size estimates by comparing estimates for 10 avalanches made by 170 avalanche professionals working in Europe or North America. In the other two studies, both completed as pilot studies, we explore reliability in the mappings of six avalanches from oblique photographs from 10 participants and the mappings of avalanches visible on 2.9 km<inline-formula><mml:math id="M1" display="inline"><mml:msup><mml:mi/><mml:mn mathvariant="normal">2</mml:mn></mml:msup></mml:math></inline-formula> of remotely sensed imagery in four different spatial resolutions from 5 participants.
We observed an average agreement of 66 % in the most frequently given avalanche size, while agreement with the avalanche size considered “correct” was 74 %. Moreover, European avalanche practitioners rated avalanches significantly larger for 8 out of 10 avalanches, compared to North Americans. Assuming that participants are equally competent in the estimation of avalanche size, we calculated a score describing the factor required to obtain the observed agreement rate between any two size estimates. This factor was 0.72 in our dataset. It can be regarded as the certainty related to a size estimate by an individual and thus provides an indication of the reliability of a label. For the outlines mapped from oblique photographs, we noted a mean overlapping proportion of 52 % for any two avalanche mappings and 60 % compared to a reference mapping. The outlines mapped from remotely sensed imagery had a mean overlapping proportion of 46 % (image resolution of 2 m) to 68 % (25 <inline-formula><mml:math id="M2" display="inline"><mml:mrow class="unit"><mml:mi mathvariant="normal">cm</mml:mi></mml:mrow></mml:math></inline-formula>) between any two mappings and 64 % (2 m) to 80 % (25 <inline-formula><mml:math id="M3" display="inline"><mml:mrow class="unit"><mml:mi mathvariant="normal">cm</mml:mi></mml:mrow></mml:math></inline-formula>) when compared to the reference.
The presented findings demonstrate that the reliability of size estimates and of mapped avalanche outlines is limited. As these data are often used as reference data or even ground truth to validate further applications, the identified limitations and uncertainties may influence results and should be considered.</p>
  </abstract>
    
<funding-group>
<award-group id="gs1">
<funding-source>Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung</funding-source>
<award-id>200021_172800</award-id>
</award-group>
</funding-group>
</article-meta>
  </front>
<body>
      

<sec id="Ch1.S1" sec-type="intro">
  <label>1</label><title>Introduction</title>
      <?pagebreak page2896?><p id="d1e184">Information on the location and size of avalanches is crucial for avalanche forecasting. A consistent and accurate documentation of the outlines of avalanches is important for applications such as avalanche-related risk management, hazard mitigation measures or hazard zone planning. In addition, these data are used as ground truth, for instance, for the validation of numerical avalanche simulations <xref ref-type="bibr" rid="bib1.bibx48" id="paren.1"><named-content content-type="pre">e.g.,</named-content></xref>, when training models for the automated detection of avalanches on satellite images <xref ref-type="bibr" rid="bib1.bibx21" id="paren.2"><named-content content-type="pre">e.g.,</named-content></xref> or for training models estimating avalanche size from snowpack simulations <xref ref-type="bibr" rid="bib1.bibx29" id="paren.3"><named-content content-type="pre">e.g.,</named-content></xref>. However, avalanche size estimates are subjective as they cannot easily be measured like, for instance, earthquakes. The same is true for avalanche outlines, where no objective way of determining them exists. In many applications where direct measurements are not possible, human estimates are used as the reference, sometimes referred to as a “gold standard” <xref ref-type="bibr" rid="bib1.bibx47" id="paren.4"><named-content content-type="pre">e.g.,</named-content></xref>. Applications, where such data are used, include mapping of landslides <xref ref-type="bibr" rid="bib1.bibx2 bib1.bibx17" id="paren.5"/>, identifying rock glaciers <xref ref-type="bibr" rid="bib1.bibx8" id="paren.6"/>, or the estimation of avalanche size and danger <xref ref-type="bibr" rid="bib1.bibx36" id="paren.7"><named-content content-type="pre">e.g.,</named-content></xref>. When these data are used for validation, errors in the estimates may cause an observed reduction in model or forecast performance, simply due to errors in the reference <xref ref-type="bibr" rid="bib1.bibx5 bib1.bibx26" id="paren.8"><named-content content-type="pre">e.g.,</named-content></xref>. Therefore, quantifying reliability, defined as the consistency of repeated measurements or judgments of the same event relying on the same process <xref ref-type="bibr" rid="bib1.bibx13" id="paren.9"/>, is vital.</p>
      <p id="d1e227">The reliability of judgments of something that cannot be known directly may be described using Brunswik's lens model <xref ref-type="bibr" rid="bib1.bibx38" id="paren.10"/>: the parameter that cannot be directly measured is estimated using the information available (data). These data may be imperfectly describing the parameter of interest. The connection between the parameter and the actual event is the accuracy of the estimate. It may be reduced by unreliability either in the information (data) or in the information processing for making the judgment <xref ref-type="bibr" rid="bib1.bibx38" id="paren.11"><named-content content-type="pre">skill of the judge;</named-content></xref>. The reliability of judgments may be approximated with repeated estimates, regression models or the measurement of agreement among estimates <xref ref-type="bibr" rid="bib1.bibx38" id="paren.12"/>. Such investigations, for comparable tasks where human estimates are important, have revealed that the automated mapping of clean glacier ice is at least as accurate as manual digitization <xref ref-type="bibr" rid="bib1.bibx34" id="paren.13"/>. <xref ref-type="bibr" rid="bib1.bibx17" id="text.14"/> found the time available for field reconnaissance to correlate with the accuracy of landslide event inventory maps. <xref ref-type="bibr" rid="bib1.bibx8" id="text.15"/> analyzed the variability in rock glacier inventories and found it to depend, in comparable proportions, on inter-operator variability and the quality of available imagery.</p>
      <p id="d1e251">Since both avalanche size and avalanche outlines are currently assessed relying on human interpreters and since consistent and accurate size estimates and avalanche outlines are key data for several applications, our objective is to quantify the reliability of these data. We expand previous studies exploring the consistency in avalanche size estimates <xref ref-type="bibr" rid="bib1.bibx33 bib1.bibx24" id="paren.16"/> using a larger sample. Moreover, we quantify the reliability in avalanche outlines mapped from oblique photographs and remotely sensed imagery and investigate potential factors explaining inter- and intra-rater variations.</p>
      <p id="d1e257">In three independent user studies we address the following two research questions:
<list list-type="order"><list-item>
      <p id="d1e262">To what degree do experts agree when rating the size of an avalanche from photographs?</p></list-item><list-item>
      <p id="d1e266">To what degree do experts agree when mapping the outline of avalanches from oblique photographs or from remotely sensed imagery?</p></list-item></list>
Moreover, we explore potential factors influencing the agreement rates in size estimates or avalanche outline mappings. This allows for the estimation of benchmark values describing the reliability of these kind of data and hence the interpretation of the performance of applications relying on these data.</p>
</sec>
<sec id="Ch1.S2">
  <label>2</label><title>Background</title>
      <p id="d1e278">Avalanche size may be assessed by installing infrastructure to measure impact pressure <xref ref-type="bibr" rid="bib1.bibx37" id="paren.17"><named-content content-type="pre">e.g.,</named-content></xref> or by determining deposit volumes with photogrammetry <xref ref-type="bibr" rid="bib1.bibx15" id="paren.18"><named-content content-type="pre">e.g.,</named-content></xref>, optionally complemented with snow density samples of the avalanche deposit or by assuming a plausible density to calculate mass <xref ref-type="bibr" rid="bib1.bibx24" id="paren.19"/>. Given current technologies, this is not feasible for all avalanches, and in addition it was not possible at all until a few years ago. Therefore most avalanche inventories rely on size estimates made by humans.
Even though avalanches may be identified in remotely sensed imagery with high locational accuracy, there is as of yet no objective way to determine the outlines of avalanches, and – so far – all automatic approaches have been validated against manual mappings <xref ref-type="bibr" rid="bib1.bibx27 bib1.bibx3" id="paren.20"><named-content content-type="pre">e.g.,</named-content></xref>. Furthermore, suitable remotely sensed imagery is often not available; therefore avalanche outlines are mostly manually mapped, directly in the field or later from photographs.</p>
      <?pagebreak page2897?><p id="d1e299">In practice, field observers or the public may provide an estimation of avalanche size together with the approximate location of the avalanche (a point) or map the outlines of avalanches, while avalanche forecasters recording avalanches may also use photographs provided by third parties for these tasks. It is common practice for avalanches to be assigned a size estimate using a scale. Standardized scales were first proposed about 60 years ago by the <xref ref-type="bibr" rid="bib1.bibx43" id="text.21"/> to “provide an effective vehicle for communication between the observers themselves and others” <xref ref-type="bibr" rid="bib1.bibx30" id="paren.22"><named-content content-type="post">p. 15</named-content></xref>. The earliest classification of avalanches into size categories was based on destructive potential <xref ref-type="bibr" rid="bib1.bibx43" id="paren.23"/>. Since then, the classification has been extended and refined by analyzing mass and frequency distributions of avalanches <xref ref-type="bibr" rid="bib1.bibx30" id="paren.24"/>. This scale was adopted in several countries (among others, Canada and New Zealand). In addition, in the United States the destructive scale is often combined with a relative scale, where avalanches are given a size relative to the avalanche path they occurred on <xref ref-type="bibr" rid="bib1.bibx4" id="paren.25"/>. In this system, the size of an avalanche is dependent on its location <xref ref-type="bibr" rid="bib1.bibx30" id="paren.26"/>. Both scales use five size classes, with size 1 being the smallest and size 5 being the largest avalanche. With some variations, the destructive scale was adopted in Europe in 2009 and later complemented with more details. An overview of the scales currently used in North America and Europe is shown in Table <xref ref-type="table" rid="Ch1.T1"/>.</p>

<table-wrap id="Ch1.T1" specific-use="star"><?xmltex \currentcnt{1}?><label>Table 1</label><caption><p id="d1e327">Definition of avalanche size for Europe <xref ref-type="bibr" rid="bib1.bibx14" id="paren.27"/> and North America (<xref ref-type="bibr" rid="bib1.bibx12 bib1.bibx1" id="altparen.28"/>; Canadian definition for the description of potential damage, the other parameters are identical). In Europe, length and volume are subsumed under the headline typical dimensions <xref ref-type="bibr" rid="bib1.bibx14" id="paren.29"/>.</p></caption><oasis:table frame="topbot"><?xmltex \begin{scaleboxenv}{.86}[.86]?><oasis:tgroup cols="4">
     <oasis:colspec colnum="1" colname="col1" align="justify" colwidth="0.4cm"/>
     <oasis:colspec colnum="2" colname="col2" align="justify" colwidth="2.9cm"/>
     <oasis:colspec colnum="3" colname="col3" align="justify" colwidth="8cm"/>
     <oasis:colspec colnum="4" colname="col4" align="justify" colwidth="7.5cm"/>
     <oasis:thead>
       <oasis:row rowsep="1">
         <oasis:entry colname="col1">Size</oasis:entry>
         <oasis:entry colname="col2">Parameter</oasis:entry>
         <oasis:entry colname="col3">European definition</oasis:entry>
         <oasis:entry colname="col4">North American definition</oasis:entry>
       </oasis:row>
     </oasis:thead>
     <oasis:tbody>
       <oasis:row>
         <oasis:entry colname="col1">1</oasis:entry>
         <oasis:entry colname="col2">Potential damage</oasis:entry>
         <oasis:entry colname="col3">Unlikely to bury a person, except in runout zones with unfavorable terrain features (e.g., terrain traps)</oasis:entry>
         <oasis:entry colname="col4">Relatively harmless to people</oasis:entry>
       </oasis:row>
       <oasis:row>
         <oasis:entry colname="col1"/>
         <oasis:entry colname="col2">Runout</oasis:entry>
         <oasis:entry colname="col3">Stops within steep slopes</oasis:entry>
         <oasis:entry colname="col4">–</oasis:entry>
       </oasis:row>
       <oasis:row>
         <oasis:entry colname="col1"/>
         <oasis:entry colname="col2">Length</oasis:entry>
         <oasis:entry colname="col3">10–30 m</oasis:entry>
         <oasis:entry colname="col4">10 m</oasis:entry>
       </oasis:row>
       <oasis:row>
         <oasis:entry colname="col1"/>
         <oasis:entry colname="col2">Volume</oasis:entry>
         <oasis:entry colname="col3">100 m<inline-formula><mml:math id="M4" display="inline"><mml:msup><mml:mi/><mml:mn mathvariant="normal">3</mml:mn></mml:msup></mml:math></inline-formula></oasis:entry>
         <oasis:entry colname="col4">–</oasis:entry>
       </oasis:row>
       <oasis:row rowsep="1">
         <oasis:entry colname="col1"/>
         <oasis:entry colname="col2">Mass/impact pressure</oasis:entry>
         <oasis:entry colname="col3">–</oasis:entry>
         <oasis:entry colname="col4"><inline-formula><mml:math id="M5" display="inline"><mml:mo>&lt;</mml:mo></mml:math></inline-formula> 10 t/1 <inline-formula><mml:math id="M6" display="inline"><mml:mrow class="unit"><mml:mi mathvariant="normal">kPa</mml:mi></mml:mrow></mml:math></inline-formula></oasis:entry>
       </oasis:row>
       <oasis:row>
         <oasis:entry colname="col1">2</oasis:entry>
         <oasis:entry colname="col2">Potential damage</oasis:entry>
         <oasis:entry colname="col3">May bury, injure or kill a person</oasis:entry>
         <oasis:entry colname="col4">Could bury, injure or kill a person</oasis:entry>
       </oasis:row>
       <oasis:row>
         <oasis:entry colname="col1"/>
         <oasis:entry colname="col2">Runout</oasis:entry>
         <oasis:entry colname="col3">May reach the end of the relevant steep slope</oasis:entry>
         <oasis:entry colname="col4">–</oasis:entry>
       </oasis:row>
       <oasis:row>
         <oasis:entry colname="col1"/>
         <oasis:entry colname="col2">Length</oasis:entry>
         <oasis:entry colname="col3">50–200 m</oasis:entry>
         <oasis:entry colname="col4">100 m</oasis:entry>
       </oasis:row>
       <oasis:row>
         <oasis:entry colname="col1"/>
         <oasis:entry colname="col2">Volume</oasis:entry>
         <oasis:entry colname="col3">1000 m<inline-formula><mml:math id="M7" display="inline"><mml:msup><mml:mi/><mml:mn mathvariant="normal">3</mml:mn></mml:msup></mml:math></inline-formula></oasis:entry>
         <oasis:entry colname="col4">–</oasis:entry>
       </oasis:row>
       <oasis:row rowsep="1">
         <oasis:entry colname="col1"/>
         <oasis:entry colname="col2">Mass/impact pressure</oasis:entry>
         <oasis:entry colname="col3">–</oasis:entry>
         <oasis:entry colname="col4">100 <inline-formula><mml:math id="M8" display="inline"><mml:mrow class="unit"><mml:mi mathvariant="normal">t</mml:mi></mml:mrow></mml:math></inline-formula>/10 <inline-formula><mml:math id="M9" display="inline"><mml:mrow class="unit"><mml:mi mathvariant="normal">kPa</mml:mi></mml:mrow></mml:math></inline-formula></oasis:entry>
       </oasis:row>
       <oasis:row>
         <oasis:entry colname="col1">3</oasis:entry>
         <oasis:entry colname="col2">Potential damage</oasis:entry>
         <oasis:entry colname="col3">May bury and destroy cars, damage trucks, destroy small buildings, and break a few trees</oasis:entry>
         <oasis:entry colname="col4">Could bury and destroy a car, damage a truck, destroy a wood-frame house, or break a few trees</oasis:entry>
       </oasis:row>
       <oasis:row>
         <oasis:entry colname="col1"/>
         <oasis:entry colname="col2">Runout</oasis:entry>
         <oasis:entry colname="col3">May cross flat terrain (well below 30<inline-formula><mml:math id="M10" display="inline"><mml:msup><mml:mi/><mml:mo>∘</mml:mo></mml:msup></mml:math></inline-formula>) over a distance of less than 50 m</oasis:entry>
         <oasis:entry colname="col4">–</oasis:entry>
       </oasis:row>
       <oasis:row>
         <oasis:entry colname="col1"/>
         <oasis:entry colname="col2">Length</oasis:entry>
         <oasis:entry colname="col3">Several hundred meters</oasis:entry>
         <oasis:entry colname="col4">1 <inline-formula><mml:math id="M11" display="inline"><mml:mrow class="unit"><mml:mi mathvariant="normal">km</mml:mi></mml:mrow></mml:math></inline-formula></oasis:entry>
       </oasis:row>
       <oasis:row>
         <oasis:entry colname="col1"/>
         <oasis:entry colname="col2">Volume</oasis:entry>
         <oasis:entry colname="col3">10 000 m<inline-formula><mml:math id="M12" display="inline"><mml:msup><mml:mi/><mml:mn mathvariant="normal">3</mml:mn></mml:msup></mml:math></inline-formula></oasis:entry>
         <oasis:entry colname="col4">–</oasis:entry>
       </oasis:row>
       <oasis:row rowsep="1">
         <oasis:entry colname="col1"/>
         <oasis:entry colname="col2">Mass/impact pressure</oasis:entry>
         <oasis:entry colname="col3">–</oasis:entry>
         <oasis:entry colname="col4">1000 <inline-formula><mml:math id="M13" display="inline"><mml:mrow class="unit"><mml:mi mathvariant="normal">t</mml:mi></mml:mrow></mml:math></inline-formula>/100 <inline-formula><mml:math id="M14" display="inline"><mml:mrow class="unit"><mml:mi mathvariant="normal">kPa</mml:mi></mml:mrow></mml:math></inline-formula></oasis:entry>
       </oasis:row>
       <oasis:row>
         <oasis:entry colname="col1">4</oasis:entry>
         <oasis:entry colname="col2">Potential damage</oasis:entry>
         <oasis:entry colname="col3">May bury and destroy trucks and trains, may destroy fairly large buildings and small areas of forest</oasis:entry>
         <oasis:entry colname="col4">Could destroy a railway car, a large truck, several buildings or a forest of approximately 4 ha</oasis:entry>
       </oasis:row>
       <oasis:row>
         <oasis:entry colname="col1"/>
         <oasis:entry colname="col2">Runout</oasis:entry>
         <oasis:entry colname="col3">Crosses flat terrain (well below 30<inline-formula><mml:math id="M15" display="inline"><mml:msup><mml:mi/><mml:mo>∘</mml:mo></mml:msup></mml:math></inline-formula>) over a distance of more than 50 m; may reach the valley floor</oasis:entry>
         <oasis:entry colname="col4">–</oasis:entry>
       </oasis:row>
       <oasis:row>
         <oasis:entry colname="col1"/>
         <oasis:entry colname="col2">Length</oasis:entry>
         <oasis:entry colname="col3">1–2 <inline-formula><mml:math id="M16" display="inline"><mml:mrow class="unit"><mml:mi mathvariant="normal">km</mml:mi></mml:mrow></mml:math></inline-formula></oasis:entry>
         <oasis:entry colname="col4">2 <inline-formula><mml:math id="M17" display="inline"><mml:mrow class="unit"><mml:mi mathvariant="normal">km</mml:mi></mml:mrow></mml:math></inline-formula></oasis:entry>
       </oasis:row>
       <oasis:row>
         <oasis:entry colname="col1"/>
         <oasis:entry colname="col2">Volume</oasis:entry>
         <oasis:entry colname="col3">100 000 m<inline-formula><mml:math id="M18" display="inline"><mml:msup><mml:mi/><mml:mn mathvariant="normal">3</mml:mn></mml:msup></mml:math></inline-formula></oasis:entry>
         <oasis:entry colname="col4">–</oasis:entry>
       </oasis:row>
       <oasis:row rowsep="1">
         <oasis:entry colname="col1"/>
         <oasis:entry colname="col2">Mass/impact pressure</oasis:entry>
         <oasis:entry colname="col3">–</oasis:entry>
         <oasis:entry colname="col4">10 000 <inline-formula><mml:math id="M19" display="inline"><mml:mrow class="unit"><mml:mi mathvariant="normal">t</mml:mi></mml:mrow></mml:math></inline-formula>/500 <inline-formula><mml:math id="M20" display="inline"><mml:mrow class="unit"><mml:mi mathvariant="normal">kPa</mml:mi></mml:mrow></mml:math></inline-formula></oasis:entry>
       </oasis:row>
       <oasis:row>
         <oasis:entry colname="col1">5</oasis:entry>
         <oasis:entry colname="col2">Potential damage</oasis:entry>
         <oasis:entry colname="col3">May devastate the landscape and has catastrophic destructive potential</oasis:entry>
         <oasis:entry colname="col4">Largest snow avalanches known; could destroy a village or a forest area of approximately 40 ha</oasis:entry>
       </oasis:row>
       <oasis:row>
         <oasis:entry colname="col1"/>
         <oasis:entry colname="col2">Runout</oasis:entry>
         <oasis:entry colname="col3">Largest avalanche known; reaches the valley floor</oasis:entry>
         <oasis:entry colname="col4">–</oasis:entry>
       </oasis:row>
       <oasis:row>
         <oasis:entry colname="col1"/>
         <oasis:entry colname="col2">Length</oasis:entry>
         <oasis:entry colname="col3"><inline-formula><mml:math id="M21" display="inline"><mml:mo>&gt;</mml:mo></mml:math></inline-formula> 2 <inline-formula><mml:math id="M22" display="inline"><mml:mrow class="unit"><mml:mi mathvariant="normal">km</mml:mi></mml:mrow></mml:math></inline-formula></oasis:entry>
         <oasis:entry colname="col4">3 <inline-formula><mml:math id="M23" display="inline"><mml:mrow class="unit"><mml:mi mathvariant="normal">km</mml:mi></mml:mrow></mml:math></inline-formula></oasis:entry>
       </oasis:row>
       <oasis:row>
         <oasis:entry colname="col1"/>
         <oasis:entry colname="col2">Volume</oasis:entry>
         <oasis:entry colname="col3"><inline-formula><mml:math id="M24" display="inline"><mml:mo>&gt;</mml:mo></mml:math></inline-formula> 100 000 m<inline-formula><mml:math id="M25" display="inline"><mml:msup><mml:mi/><mml:mn mathvariant="normal">3</mml:mn></mml:msup></mml:math></inline-formula></oasis:entry>
         <oasis:entry colname="col4">–</oasis:entry>
       </oasis:row>
       <oasis:row>
         <oasis:entry colname="col1"/>
         <oasis:entry colname="col2">Mass/impact pressure</oasis:entry>
         <oasis:entry colname="col3">–</oasis:entry>
         <oasis:entry colname="col4">100 000 <inline-formula><mml:math id="M26" display="inline"><mml:mrow class="unit"><mml:mi mathvariant="normal">t</mml:mi></mml:mrow></mml:math></inline-formula>/1000 <inline-formula><mml:math id="M27" display="inline"><mml:mrow class="unit"><mml:mi mathvariant="normal">kPa</mml:mi></mml:mrow></mml:math></inline-formula></oasis:entry>
       </oasis:row>
     </oasis:tbody>
   </oasis:tgroup><?xmltex \end{scaleboxenv}?></oasis:table><?xmltex \gdef\@currentlabel{1}?></table-wrap>

      <p id="d1e911">Inventories of avalanches mapped either directly in the field or later from photographs have been used in numerous studies <xref ref-type="bibr" rid="bib1.bibx20 bib1.bibx10 bib1.bibx42" id="paren.30"><named-content content-type="pre">e.g.,</named-content></xref> but are known to be incomplete <xref ref-type="bibr" rid="bib1.bibx36" id="paren.31"/> and biased towards accessible terrain and larger avalanches <xref ref-type="bibr" rid="bib1.bibx23" id="paren.32"/>. Avalanche size may be directly derived from avalanche outlines <xref ref-type="bibr" rid="bib1.bibx36 bib1.bibx46 bib1.bibx9" id="paren.33"/>, for example, by thresholding the mapped area. In addition to manual avalanche outline mapping, avalanches have increasingly been mapped (manually or automatically) from remotely sensed imagery such as satellite images or orthophotos acquired from airplanes or drones <xref ref-type="bibr" rid="bib1.bibx25 bib1.bibx16 bib1.bibx9 bib1.bibx3 bib1.bibx21" id="paren.34"><named-content content-type="pre">e.g.,</named-content></xref>. Specifically satellite imagery has the potential to close the information gap in avalanche documentation and record avalanche occurrences over large areas with a consistent methodology, thereby complementing existing databases <xref ref-type="bibr" rid="bib1.bibx27 bib1.bibx45 bib1.bibx16 bib1.bibx9" id="paren.35"><named-content content-type="pre">e.g.,</named-content></xref>.</p>
</sec>
<sec id="Ch1.S3">
  <label>3</label><title>Data and methods</title>
      <p id="d1e947">To explore the reliability of avalanche size estimates and avalanche outline determination, we conducted three user studies, described in Sect. <xref ref-type="sec" rid="Ch1.S3.SS1"/> to <xref ref-type="sec" rid="Ch1.S3.SS3"/>. In all three studies, we simulated different typical size estimation or avalanche mapping tasks based on either oblique photos or remotely sensed images. For each of the three experiments, this translated to the following common task for the study participants: an assessor is given an image and has to (1) detect the avalanche(s) in the image. If an avalanche is detected, the assessor (2) either judges the size of the avalanche or distinguishes between an avalanche and no avalanche by drawing an outline.</p>
<sec id="Ch1.S3.SS1">
  <label>3.1</label><title>Study 1: avalanche size estimation</title>
      <p id="d1e961">To explore the reliability of avalanche size estimates provided by avalanche practitioners, we developed a survey consisting of 10 photographs of avalanches (see the Supplement). The photographs used were originally captured for the purpose of documenting avalanches. They show clearly identifiable avalanches in various terrain and were chosen to provide a diverse set of perspectives, sizes and illumination, as well as the presence of various reference objects.
In the survey, each participant was asked to estimate the size of each of the 10 avalanches using the five-class integer scale, where we refer to estimates as “full” sizes (for instance, size 3; see Table <xref ref-type="table" rid="Ch1.T1"/>). After estimating the full size of an avalanche, participants had the opportunity to provide an intermediate size (“half” size, nine levels; for instance, size 2.5). As a second task, we asked participants to rate the importance of the factors characterizing avalanche size for their size estimations on a four-point Likert scale as either <italic>very important</italic>, <italic>important</italic>, <italic>less important</italic> or <italic>not at all important</italic> (factors: destructive potential, dimensions, runout and volume; Table <xref ref-type="table" rid="Ch1.T1"/>). We designed the survey with a European audience in mind and only later decided to extend it to North America. For this reason, runout and volume were included as factors even though they are not part of the North American avalanche size definition. Similarly, in the European definition typical length and volume are presented under the headline typical dimensions <xref ref-type="bibr" rid="bib1.bibx14" id="paren.36"/>, a term which is not present in the North American definition.</p>
      <p id="d1e984">The survey was sent to avalanche practitioners, primarily regional avalanche forecasters in Europe and North America, through personal contacts or using forecast center mailing lists. The survey was available in English, French, German and Italian. We asked participants at the beginning of the survey whether they were avalanche forecasters and in which country they work. In total, 170 responses were received: 105 from Europe and 65 from North America. The proportion of professional avalanche forecasters in our dataset was 86 % (146). The other 24 participants either had additional roles besides forecasting or worked closely with the avalanche warning service, for example as avalanche educators, mountain guides, ski patrollers or field observers for a warning service. The professional forecasters in our sample were from the United States (39 participants), Italy (33), Canada (17), Norway (15), Spain (10), Austria (10) and Switzerland (7), while all other countries had 2 or fewer participants.</p>
</sec>
<sec id="Ch1.S3.SS2">
  <label>3.2</label><title>Study 2: avalanche mapping from oblique photographs</title>
      <p id="d1e995">To investigate the reliability in avalanche outlines, we asked nine people, who map avalanches as part of their professional duties, to map six avalanches in the area around Davos (Switzerland) from winter 2020/21.</p>
      <p id="d1e998">For each avalanche, we provided three to six photographs and indicated the approximate location by giving the name of a ridge or summit in the proximity of the avalanche (distance 50 to 300 m). Mapping was conducted using operational mapping tools, which provide the user with a topographic map <xref ref-type="bibr" rid="bib1.bibx39" id="paren.37"><named-content content-type="pre">at best with a scale of <inline-formula><mml:math id="M28" display="inline"><mml:mrow><mml:mn mathvariant="normal">1</mml:mn><mml:mo>:</mml:mo><mml:mn mathvariant="normal">10</mml:mn><mml:mspace linebreak="nobreak" width="0.125em"/><mml:mn mathvariant="normal">000</mml:mn></mml:mrow></mml:math></inline-formula>;</named-content></xref>, orthophotos <xref ref-type="bibr" rid="bib1.bibx40" id="paren.38"><named-content content-type="pre">resolution of 10<inline-formula><mml:math id="M29" display="inline"><mml:mo>×</mml:mo></mml:math></inline-formula>10 <inline-formula><mml:math id="M30" display="inline"><mml:mrow class="unit"><mml:mi mathvariant="normal">cm</mml:mi></mml:mrow></mml:math></inline-formula>;</named-content></xref> and slope incline classes for areas steeper than 30<inline-formula><mml:math id="M31" display="inline"><mml:msup><mml:mi/><mml:mo>∘</mml:mo></mml:msup></mml:math></inline-formula> (resolution 10 <inline-formula><mml:math id="M32" display="inline"><mml:mo>×</mml:mo></mml:math></inline-formula> 10 m). Each participant was asked to map the six avalanches with the same accuracy as they usually would when mapping avalanches. In addition to the nine participants, we used the avalanche outlines that were initially mapped for documentation purposes in winter 2020/21.</p>
      <?pagebreak page2898?><p id="d1e1057">To create a reference for the map analysis, we georeferenced one image per avalanche with the monoplotting tool developed by <xref ref-type="bibr" rid="bib1.bibx6 bib1.bibx7" id="text.39"/>, then drew and exported the avalanche outlines. Since this approach allows for a much more accurate localization of avalanche outlines, we used these as a reference in this study. For one avalanche (Fig. <xref ref-type="fig" rid="Ch1.F7"/>a), the deposit was obscured by a tree in the only photograph where the whole avalanche was visible. This part of the avalanche was therefore disregarded in the analyses including this reference.</p>
</sec>
<sec id="Ch1.S3.SS3">
  <label>3.3</label><title>Study 3: avalanche mapping from remotely sensed imagery</title>
      <p id="d1e1073">The third experiment concerned the mapping of avalanche outlines from remotely sensed imagery. In addition to comparing mapped avalanche outlines between individuals, this experiment allowed for exploring some of the potential factors influencing the quality of mapped avalanche outlines <xref ref-type="bibr" rid="bib1.bibx21" id="paren.40"><named-content content-type="pre">illumination, snow conditions, avalanche type, image resolution;</named-content></xref>.</p>
      <p id="d1e1081">We selected two georeferenced images acquired under different snow conditions (see Table <xref ref-type="table" rid="Ch1.T2"/>) without artifacts, without saturation and as 16-bit radiometric information. The images were processed in Agisoft Metashape. To obtain different resolutions, we bi-linearly resampled the data in the red–green–blue channel (RGB) to 25 <inline-formula><mml:math id="M33" display="inline"><mml:mrow class="unit"><mml:mi mathvariant="normal">cm</mml:mi></mml:mrow></mml:math></inline-formula>, 50 <inline-formula><mml:math id="M34" display="inline"><mml:mrow class="unit"><mml:mi mathvariant="normal">cm</mml:mi></mml:mrow></mml:math></inline-formula>, 1 m and 2 m spatial resolution (for native resolutions, see Table <xref ref-type="table" rid="Ch1.T2"/>). For separating illuminated from shaded areas, we used a support vector machine classifier to calculate a shadow mask <xref ref-type="bibr" rid="bib1.bibx21" id="paren.41"><named-content content-type="pre">see</named-content><named-content content-type="post">for details</named-content></xref>.</p>

<?xmltex \floatpos{t}?><table-wrap id="Ch1.T2" specific-use="star"><?xmltex \currentcnt{2}?><label>Table 2</label><caption><p id="d1e1114">Properties of remotely sensed imagery, which was used to investigate variations in the performed avalanche mapping.</p></caption><oasis:table frame="topbot"><oasis:tgroup cols="5">
     <oasis:colspec colnum="1" colname="col1" align="left"/>
     <oasis:colspec colnum="2" colname="col2" align="left"/>
     <oasis:colspec colnum="3" colname="col3" align="left"/>
     <oasis:colspec colnum="4" colname="col4" align="right"/>
     <oasis:colspec colnum="5" colname="col5" align="left"/>
     <oasis:thead>
       <oasis:row>
         <oasis:entry colname="col1">Acquisition</oasis:entry>
         <oasis:entry colname="col2">Sensor</oasis:entry>
         <oasis:entry colname="col3">Mean ground sampling</oasis:entry>
         <oasis:entry colname="col4">Area covered</oasis:entry>
         <oasis:entry colname="col5">Snow and</oasis:entry>
       </oasis:row>
       <oasis:row rowsep="1">
         <oasis:entry colname="col1">date</oasis:entry>
         <oasis:entry colname="col2"/>
         <oasis:entry colname="col3">distance (GSD)</oasis:entry>
         <oasis:entry colname="col4"> (km<inline-formula><mml:math id="M35" display="inline"><mml:msup><mml:mi/><mml:mn mathvariant="normal">2</mml:mn></mml:msup></mml:math></inline-formula>)</oasis:entry>
         <oasis:entry colname="col5">avalanche conditions</oasis:entry>
       </oasis:row>
     </oasis:thead>
     <oasis:tbody>
       <oasis:row>
         <oasis:entry colname="col1">16 March 2019</oasis:entry>
         <oasis:entry colname="col2">UltraCam Eagle M3</oasis:entry>
         <oasis:entry colname="col3">12 <inline-formula><mml:math id="M36" display="inline"><mml:mrow class="unit"><mml:mi mathvariant="normal">cm</mml:mi></mml:mrow></mml:math></inline-formula></oasis:entry>
         <oasis:entry colname="col4">2.2</oasis:entry>
         <oasis:entry colname="col5">Following a period with numerous</oasis:entry>
       </oasis:row>
       <oasis:row rowsep="1">
         <oasis:entry colname="col1"/>
         <oasis:entry colname="col2">(piloted airplane)</oasis:entry>
         <oasis:entry colname="col3"/>
         <oasis:entry colname="col4"/>
         <oasis:entry colname="col5">dry-snow avalanches</oasis:entry>
       </oasis:row>
       <oasis:row>
         <oasis:entry colname="col1">25 February 2021</oasis:entry>
         <oasis:entry colname="col2">WingtraOne (drone)</oasis:entry>
         <oasis:entry colname="col3">4 <inline-formula><mml:math id="M37" display="inline"><mml:mrow class="unit"><mml:mi mathvariant="normal">cm</mml:mi></mml:mrow></mml:math></inline-formula></oasis:entry>
         <oasis:entry colname="col4">0.7</oasis:entry>
         <oasis:entry colname="col5">Following a period with numerous</oasis:entry>
       </oasis:row>
       <oasis:row>
         <oasis:entry colname="col1"/>
         <oasis:entry colname="col2"/>
         <oasis:entry colname="col3"/>
         <oasis:entry colname="col4"/>
         <oasis:entry colname="col5">wet-snow avalanches</oasis:entry>
       </oasis:row>
     </oasis:tbody>
   </oasis:tgroup></oasis:table><?xmltex \gdef\@currentlabel{2}?></table-wrap>

      <?pagebreak page2899?><p id="d1e1263">We provided a standardized introduction to the five participants, who were all familiar with avalanches and remotely sensed imagery. All visible avalanches were to be digitized in the software ArcGIS Pro, starting from the coarsest (2 m) and ending with the finest resolution (25 <inline-formula><mml:math id="M38" display="inline"><mml:mrow class="unit"><mml:mi mathvariant="normal">cm</mml:mi></mml:mrow></mml:math></inline-formula>). Images with higher resolution were only made available after the mapping of the (one-step) coarser resolution had been completed. Participants could not re-examine their earlier mappings. They had access to the topographic map <xref ref-type="bibr" rid="bib1.bibx39" id="paren.42"><named-content content-type="pre">at best a scale of <inline-formula><mml:math id="M39" display="inline"><mml:mrow><mml:mn mathvariant="normal">1</mml:mn><mml:mo>:</mml:mo><mml:mn mathvariant="normal">10</mml:mn><mml:mspace width="0.125em" linebreak="nobreak"/><mml:mn mathvariant="normal">000</mml:mn></mml:mrow></mml:math></inline-formula>;</named-content></xref> and slope incline classes for areas steeper than 30<inline-formula><mml:math id="M40" display="inline"><mml:msup><mml:mi/><mml:mo>∘</mml:mo></mml:msup></mml:math></inline-formula> (resolution of 10 <inline-formula><mml:math id="M41" display="inline"><mml:mo>×</mml:mo></mml:math></inline-formula> 10 m). We instructed participants to outline all areas with signs of avalanche activity rather than drawing individual events; thus, they were asked to delimit all visible avalanche regions but not to separate them into individual avalanche polygons. The participants did not see the mapped outlines from other participants before they had finished with the highest resolution.</p>
</sec>
<sec id="Ch1.S3.SS4">
  <label>3.4</label><title>Data analysis</title>
<sec id="Ch1.S3.SS4.SSS1">
  <label>3.4.1</label><title>Avalanche size estimates</title>
      <p id="d1e1326">Presumably, having many assessors performing the same task is a rare exception; thus, in most situations only a single estimate for avalanche size is available. Therefore, the reliability of an individual estimate is of interest. Not making an assumption about whether any two size estimates contain the true label, the agreement between raters can be considered an indirect indicator of reliability <xref ref-type="bibr" rid="bib1.bibx38" id="paren.43"/>. For the avalanche size estimation study (Sect. <xref ref-type="sec" rid="Ch1.S3.SS1"/>), we calculated inter-rater agreement as the proportion of agreements in avalanche size between any two raters for the 10 avalanches (<inline-formula><mml:math id="M42" display="inline"><mml:mrow><mml:msub><mml:mi>P</mml:mi><mml:mi mathvariant="normal">agree</mml:mi></mml:msub></mml:mrow></mml:math></inline-formula>). Following <xref ref-type="bibr" rid="bib1.bibx38" id="text.44"/>, if random errors between two raters are independent, then the correlation between two raters' estimates cannot be larger than the product of their reliabilities, except by chance. In other words, without knowing which rater is more competent or reliable, the reliability (rel) of an individual rater is the geometric mean of the individual reliabilities <xref ref-type="bibr" rid="bib1.bibx41" id="paren.45"><named-content content-type="post">p. 35</named-content></xref>. In the special case with two raters <inline-formula><mml:math id="M43" display="inline"><mml:mrow><mml:mi>i</mml:mi><mml:mo>=</mml:mo><mml:mo mathvariant="italic">{</mml:mo><mml:mn mathvariant="normal">1</mml:mn><mml:mo>,</mml:mo><mml:mn mathvariant="normal">2</mml:mn><mml:mo mathvariant="italic">}</mml:mo></mml:mrow></mml:math></inline-formula>, rel can be derived as
              <disp-formula id="Ch1.E1" content-type="numbered"><label>1</label><mml:math id="M44" display="block"><mml:mrow><mml:mi mathvariant="normal">rel</mml:mi><mml:mo>=</mml:mo><mml:msqrt><mml:mrow><mml:mi mathvariant="normal">rel</mml:mi><mml:mo>(</mml:mo><mml:mn mathvariant="normal">1</mml:mn><mml:mo>)</mml:mo><mml:mo>×</mml:mo><mml:mi mathvariant="normal">rel</mml:mi><mml:mo>(</mml:mo><mml:mn mathvariant="normal">2</mml:mn><mml:mo>)</mml:mo></mml:mrow></mml:msqrt><mml:mo>=</mml:mo><mml:msqrt><mml:mrow><mml:msub><mml:mi>P</mml:mi><mml:mi mathvariant="normal">agree</mml:mi></mml:msub><mml:mo>(</mml:mo><mml:mn mathvariant="normal">1</mml:mn><mml:mo>,</mml:mo><mml:mn mathvariant="normal">2</mml:mn><mml:mo>)</mml:mo></mml:mrow></mml:msqrt><mml:mo>.</mml:mo></mml:mrow></mml:math></disp-formula>
            Reliability rel thus provides an indication regarding the certainty related to estimates by individuals <xref ref-type="bibr" rid="bib1.bibx38" id="paren.46"><named-content content-type="post">pp. 84–85</named-content></xref>.</p>
      <p id="d1e1430">Several studies have shown that the competency of raters influences the reliability of the labels <xref ref-type="bibr" rid="bib1.bibx26 bib1.bibx49" id="paren.47"><named-content content-type="pre">e.g.,</named-content></xref>. We therefore investigated whether some raters more often provided different or rather extreme size estimates compared to others. As we are lacking an independent ground truth label, we infer a ground truth size assuming that the consensus or most frequently chosen size is a suitable approximation. This is a frequently used approach when no ground truth label is available <xref ref-type="bibr" rid="bib1.bibx26 bib1.bibx49" id="paren.48"><named-content content-type="pre">e.g.,</named-content></xref>. Thus, we extracted the mode (<inline-formula><mml:math id="M45" display="inline"><mml:mrow><mml:msub><mml:mi>s</mml:mi><mml:mi mathvariant="normal">mode</mml:mi></mml:msub></mml:mrow></mml:math></inline-formula>) and median size. However, as the number of participants differed between North America and Europe, and not wanting to favor either in case of differences, we considered the mean of the corresponding median size in North America and Europe for avalanche <inline-formula><mml:math id="M46" display="inline"><mml:mi>j</mml:mi></mml:math></inline-formula> as the reference size <inline-formula><mml:math id="M47" display="inline"><mml:mrow><mml:msub><mml:mover accent="true"><mml:mi>s</mml:mi><mml:mo mathvariant="normal">‾</mml:mo></mml:mover><mml:mi>j</mml:mi></mml:msub></mml:mrow></mml:math></inline-formula>. In the case that <inline-formula><mml:math id="M48" display="inline"><mml:mrow><mml:msub><mml:mover accent="true"><mml:mi>s</mml:mi><mml:mo mathvariant="normal">‾</mml:mo></mml:mover><mml:mi>j</mml:mi></mml:msub></mml:mrow></mml:math></inline-formula> was between two integer values, for instance, 2.5, we considered the result inconclusive and treated the correspondingly lower and higher integer size as correct too (here size 2 and 3). Similarly, in the case of equal votes for two avalanche sizes, we considered both for the calculation of agreement with <inline-formula><mml:math id="M49" display="inline"><mml:mrow><mml:msub><mml:mi>s</mml:mi><mml:mi mathvariant="normal">mode</mml:mi></mml:msub></mml:mrow></mml:math></inline-formula>.</p>
      <p id="d1e1501">To obtain an indication on the competency of individual raters, we derived a proportion of those that are “correct” for each rater <inline-formula><mml:math id="M50" display="inline"><mml:mi>i</mml:mi></mml:math></inline-formula> of <inline-formula><mml:math id="M51" display="inline"><mml:mrow><mml:msub><mml:mi>P</mml:mi><mml:mi mathvariant="normal">correct</mml:mi></mml:msub></mml:mrow></mml:math></inline-formula>, defined as the number of size estimates <inline-formula><mml:math id="M52" display="inline"><mml:mrow><mml:msub><mml:mi>s</mml:mi><mml:mrow><mml:mi>i</mml:mi><mml:mi>j</mml:mi></mml:mrow></mml:msub></mml:mrow></mml:math></inline-formula> being equal to <inline-formula><mml:math id="M53" display="inline"><mml:mrow><mml:msub><mml:mover accent="true"><mml:mi>s</mml:mi><mml:mo mathvariant="normal">‾</mml:mo></mml:mover><mml:mi>j</mml:mi></mml:msub></mml:mrow></mml:math></inline-formula> divided by the number of avalanches. As an alternative approach, we calculated <inline-formula><mml:math id="M54" display="inline"><mml:mrow><mml:msub><mml:mi>P</mml:mi><mml:mi mathvariant="normal">mode</mml:mi></mml:msub></mml:mrow></mml:math></inline-formula> for each rater <inline-formula><mml:math id="M55" display="inline"><mml:mi>i</mml:mi></mml:math></inline-formula>, specified as the number of size estimates <inline-formula><mml:math id="M56" display="inline"><mml:mrow><mml:msub><mml:mi>s</mml:mi><mml:mrow><mml:mi>i</mml:mi><mml:mi>j</mml:mi></mml:mrow></mml:msub></mml:mrow></mml:math></inline-formula> equal to the most frequently chosen size <inline-formula><mml:math id="M57" display="inline"><mml:mrow><mml:msub><mml:mi>s</mml:mi><mml:mi mathvariant="normal">mode</mml:mi></mml:msub></mml:mrow></mml:math></inline-formula> divided by the number of avalanches.</p>
      <p id="d1e1594">We used the Wilcoxon rank-sum test and the proportion test <xref ref-type="bibr" rid="bib1.bibx35" id="paren.49"><named-content content-type="pre">as implemented in</named-content></xref> to test for significant differences between groups. We considered <inline-formula><mml:math id="M58" display="inline"><mml:mi>p</mml:mi></mml:math></inline-formula> values <inline-formula><mml:math id="M59" display="inline"><mml:mo>≤</mml:mo></mml:math></inline-formula> 0.05 as statistically significant.</p>
</sec>
<sec id="Ch1.S3.SS4.SSS2">
  <label>3.4.2</label><title>Avalanche outline determination</title>
      <p id="d1e1624">For the outline determination exercises (studies 2 and 3), we calculated the intersection over union (IoU) as an indicator of spatial agreement in the mappings by any two annotators <xref ref-type="bibr" rid="bib1.bibx28" id="paren.50"><named-content content-type="pre">e.g.,</named-content></xref>. Here, IoU describes the overlapping proportion of two avalanche areas (AoO) relative to the combined area of the two avalanche areas (AoU):
              <disp-formula id="Ch1.E2" content-type="numbered"><label>2</label><mml:math id="M60" display="block"><mml:mrow><mml:mi mathvariant="normal">IoU</mml:mi><mml:mo>=</mml:mo><mml:mstyle displaystyle="true"><mml:mfrac style="display"><mml:mrow><mml:mi mathvariant="normal">area</mml:mi><mml:mspace width="0.25em" linebreak="nobreak"/><mml:mi mathvariant="normal">of</mml:mi><mml:mspace linebreak="nobreak" width="0.25em"/><mml:mi mathvariant="normal">overlap</mml:mi><mml:mspace linebreak="nobreak" width="0.25em"/><mml:mo>(</mml:mo><mml:mi mathvariant="normal">AoO</mml:mi><mml:mo>)</mml:mo></mml:mrow><mml:mrow><mml:mi mathvariant="normal">area</mml:mi><mml:mspace width="0.25em" linebreak="nobreak"/><mml:mi mathvariant="normal">of</mml:mi><mml:mspace linebreak="nobreak" width="0.25em"/><mml:mi mathvariant="normal">union</mml:mi><mml:mspace linebreak="nobreak" width="0.25em"/><mml:mo>(</mml:mo><mml:mi mathvariant="normal">AoU</mml:mi><mml:mo>)</mml:mo></mml:mrow></mml:mfrac></mml:mstyle><mml:mo>,</mml:mo></mml:mrow></mml:math></disp-formula>
            where IoU lies between 0 (no overlap) and 1 (full overlap; Fig. <xref ref-type="fig" rid="Ch1.F1"/>).</p>

      <?xmltex \floatpos{t}?><fig id="Ch1.F1"><?xmltex \currentcnt{1}?><?xmltex \def\figurename{Figure}?><label>Figure 1</label><caption><p id="d1e1682">Intersection over union (IoU) with the area of overlap (AoO) and the area of union (AoU).</p></caption>
            <?xmltex \igopts{width=170.716535pt}?><graphic xlink:href="https://nhess.copernicus.org/articles/23/2895/2023/nhess-23-2895-2023-f01.png"/>

          </fig>

      <p id="d1e1691">We used three variations for IoU:
<list list-type="bullet"><list-item>
      <p id="d1e1696">IoU<inline-formula><mml:math id="M61" display="inline"><mml:msub><mml:mi/><mml:mi mathvariant="normal">pairwise</mml:mi></mml:msub></mml:math></inline-formula>, which is the ratio between the intersection of any two individual mappings to the union of these two mappings;</p></list-item><list-item>
      <p id="d1e1709">IoU<inline-formula><mml:math id="M62" display="inline"><mml:msub><mml:mi/><mml:mi mathvariant="normal">all</mml:mi></mml:msub></mml:math></inline-formula>, which is the ratio between the intersection of an individual mapping to the union of all mappings;</p></list-item><list-item>
      <p id="d1e1722">IoU<inline-formula><mml:math id="M63" display="inline"><mml:msub><mml:mi/><mml:mi mathvariant="normal">ref</mml:mi></mml:msub></mml:math></inline-formula>, which is the ratio between the intersection of an individual mapping and the reference mapping to the union of these two mappings.</p></list-item></list></p>
      <p id="d1e1735">As for the avalanche size estimation study (Sect. <xref ref-type="sec" rid="Ch1.S3.SS1"/>), we explored annotator competence. In study 2 (Sect. <xref ref-type="sec" rid="Ch1.S3.SS2"/>), we used the reference mapping as ground truth. In study 3 (Sect. <xref ref-type="sec" rid="Ch1.S3.SS3"/>), with five participants, we assumed that the area marked as an avalanche by a simple majority (three out of five participants) represented a good approximation of a ground truth.</p>
</sec>
</sec>
</sec>
<?pagebreak page2900?><sec id="Ch1.S4">
  <label>4</label><title>Results</title>
<sec id="Ch1.S4.SS1">
  <label>4.1</label><title>Avalanche size estimation (study 1)</title>
      <p id="d1e1761">A total of 170 people participated in the survey and estimated the size of 10 avalanches, shown in Fig. <xref ref-type="fig" rid="Ch1.F4"/>.
The <italic>agreement rate between any two size estimates</italic> <inline-formula><mml:math id="M64" display="inline"><mml:mrow><mml:msub><mml:mi>P</mml:mi><mml:mi mathvariant="normal">agree</mml:mi></mml:msub></mml:mrow></mml:math></inline-formula> was 0.53, ranging from 0.22 to 0.68 for individual raters. Nine of these raters had an agreement rate lower than the 95th percentile of the 170 participants (<inline-formula><mml:math id="M65" display="inline"><mml:mrow><mml:msub><mml:mi>P</mml:mi><mml:mi mathvariant="normal">agree</mml:mi></mml:msub><mml:mo>≤</mml:mo><mml:mn mathvariant="normal">0.39</mml:mn></mml:mrow></mml:math></inline-formula>), indicating particularly low correspondence with avalanche size as perceived by others. Each of these nine raters suggested at least for one avalanche a rather “extreme” avalanche size, a size which less than 10 % of the participants had chosen. Without these nine raters, the agreement rate was 0.54, which is only marginally higher than the overall agreement. Considering all responses, the mean reliability rel of individual estimates was 0.72, ranging from 0.47 to 0.82, and, if excluding the nine raters with the lowest agreement with others, rel was 0.73.</p>
      <p id="d1e1795">On average, the <italic>agreement with the size considered correct</italic> (<inline-formula><mml:math id="M66" display="inline"><mml:mrow><mml:msub><mml:mover accent="true"><mml:mi>s</mml:mi><mml:mo mathvariant="normal">‾</mml:mo></mml:mover><mml:mi>j</mml:mi></mml:msub></mml:mrow></mml:math></inline-formula>) was <inline-formula><mml:math id="M67" display="inline"><mml:mrow><mml:msub><mml:mi>P</mml:mi><mml:mi mathvariant="normal">correct</mml:mi></mml:msub><mml:mo>=</mml:mo></mml:mrow></mml:math></inline-formula> 0.74, or, if treating a simple most frequent vote <inline-formula><mml:math id="M68" display="inline"><mml:mrow><mml:msub><mml:mi>s</mml:mi><mml:mi mathvariant="normal">mode</mml:mi></mml:msub></mml:mrow></mml:math></inline-formula> as the reference size, <inline-formula><mml:math id="M69" display="inline"><mml:mrow><mml:msub><mml:mi>P</mml:mi><mml:mi mathvariant="normal">mode</mml:mi></mml:msub></mml:mrow></mml:math></inline-formula> was 0.66; 16 participants were in full agreement (<inline-formula><mml:math id="M70" display="inline"><mml:mrow><mml:msub><mml:mi>P</mml:mi><mml:mi mathvariant="normal">correct</mml:mi></mml:msub><mml:mo>=</mml:mo><mml:mn mathvariant="normal">1</mml:mn></mml:mrow></mml:math></inline-formula>) with the avalanche sizes considered the most likely size <inline-formula><mml:math id="M71" display="inline"><mml:mrow><mml:msub><mml:mover accent="true"><mml:mi>s</mml:mi><mml:mo mathvariant="normal">‾</mml:mo></mml:mover><mml:mi>j</mml:mi></mml:msub></mml:mrow></mml:math></inline-formula>, while the nine raters with the lowest agreement with others also had low values of agreement with <inline-formula><mml:math id="M72" display="inline"><mml:mrow><mml:msub><mml:mover accent="true"><mml:mi>s</mml:mi><mml:mo mathvariant="normal">‾</mml:mo></mml:mover><mml:mi>j</mml:mi></mml:msub></mml:mrow></mml:math></inline-formula>. Excluding these resulted in <inline-formula><mml:math id="M73" display="inline"><mml:mrow><mml:msub><mml:mi>P</mml:mi><mml:mi mathvariant="normal">correct</mml:mi></mml:msub><mml:mo>=</mml:mo></mml:mrow></mml:math></inline-formula> 0.76.</p>
      <p id="d1e1907">In addition to the 66 % of the respondents who provided the same size estimate as the most frequently chosen size <inline-formula><mml:math id="M74" display="inline"><mml:mrow><mml:msub><mml:mi>s</mml:mi><mml:mi mathvariant="normal">mode</mml:mi></mml:msub></mml:mrow></mml:math></inline-formula>, another 29 % chose the second-most popular neighboring size. Thus, in total 92 % of all estimates fell into two adjacent size classes highlighting that there was a reasonable consensus on the most likely size(s). Relaxing the definition for agreement even more <xref ref-type="bibr" rid="bib1.bibx33" id="paren.51"><named-content content-type="pre">as in</named-content></xref>, 97 % of the responses were size <inline-formula><mml:math id="M75" display="inline"><mml:mrow><mml:msub><mml:mi>s</mml:mi><mml:mi mathvariant="normal">mode</mml:mi></mml:msub><mml:mo>±</mml:mo><mml:mn mathvariant="normal">1</mml:mn></mml:mrow></mml:math></inline-formula>, ranging from 46 % for Fig. <xref ref-type="fig" rid="Ch1.F4"/>b to 98 % for  Fig. <xref ref-type="fig" rid="Ch1.F4"/>f. The average number of different full size classes chosen was 3.7, ranging between two for Fig. <xref ref-type="fig" rid="Ch1.F4"/>f and five for Fig. <xref ref-type="fig" rid="Ch1.F4"/>j. The latter example means that each of the five size classes were indicated at least once. This shows that even though most votes were in correspondence with one of the two most frequent size classes, at least some estimates regularly deviated strongly from this opinion.</p>
      <p id="d1e1950">An intermediate size class was given in 26 % of all cases. The agreement of the intermediate size estimated by a respondent with the most frequently indicated intermediate size <inline-formula><mml:math id="M76" display="inline"><mml:mrow><mml:msub><mml:mi>s</mml:mi><mml:mrow><mml:mi mathvariant="normal">mode</mml:mi><mml:mo>.</mml:mo><mml:mi mathvariant="normal">intermediate</mml:mi></mml:mrow></mml:msub></mml:mrow></mml:math></inline-formula> was 0.49, and for <inline-formula><mml:math id="M77" display="inline"><mml:mrow><mml:msub><mml:mi>s</mml:mi><mml:mrow><mml:mi mathvariant="normal">mode</mml:mi><mml:mo>.</mml:mo><mml:mi mathvariant="normal">intermediate</mml:mi></mml:mrow></mml:msub><mml:mo>±</mml:mo><mml:mn mathvariant="normal">0.5</mml:mn></mml:mrow></mml:math></inline-formula> the agreement was 0.74, while <inline-formula><mml:math id="M78" display="inline"><mml:mrow><mml:msub><mml:mi>P</mml:mi><mml:mi mathvariant="normal">correct</mml:mi></mml:msub></mml:mrow></mml:math></inline-formula> was 0.53. The most frequent intermediate size was always between the two most frequent full sizes, underlining that a share of participants differed in their estimates less than a full size (Fig. <xref ref-type="fig" rid="Ch1.F4"/>). The mean agreement rate <inline-formula><mml:math id="M79" display="inline"><mml:mrow><mml:msub><mml:mi>P</mml:mi><mml:mi mathvariant="normal">agree</mml:mi></mml:msub></mml:mrow></mml:math></inline-formula>, when allowing for full and intermediate sizes, was 0.37; the reliability rel was consequently 0.61.</p>

      <?xmltex \floatpos{p}?><fig id="Ch1.F2" specific-use="star"><?xmltex \currentcnt{2}?><?xmltex \def\figurename{Figure}?><label>Figure 2</label><caption><p id="d1e2016"> </p></caption>
          <?xmltex \igopts{width=341.433071pt}?><graphic xlink:href="https://nhess.copernicus.org/articles/23/2895/2023/nhess-23-2895-2023-f02-part01.jpg"/>

        </fig>

      <?xmltex \floatpos{p}?><fig id="Ch1.F3" specific-use="star"><?xmltex \currentcnt{2}?><?xmltex \def\figurename{Figure}?><label>Figure 2</label><caption><p id="d1e2027"> </p></caption>
          <?xmltex \igopts{width=341.433071pt}?><graphic xlink:href="https://nhess.copernicus.org/articles/23/2895/2023/nhess-23-2895-2023-f02-part02.jpg"/>

        </fig>

      <?xmltex \floatpos{t}?><fig id="Ch1.F4" specific-use="star"><?xmltex \currentcnt{2}?><?xmltex \def\figurename{Figure}?><label>Figure 2</label><caption><p id="d1e2038">Distribution of the size classes and the intermediate sizes assigned to avalanches/pictures <bold>(a)</bold> to <bold>(j)</bold> in the survey.</p></caption>
          <?xmltex \igopts{width=341.433071pt}?><graphic xlink:href="https://nhess.copernicus.org/articles/23/2895/2023/nhess-23-2895-2023-f02-part03.jpg"/>

        </fig>

      <p id="d1e2053">To explore if the size of an avalanche relative within an image and in relation to the surroundings influences size estimation, we included one avalanche twice though the image was cropped and flipped (avalanches in Fig. <xref ref-type="fig" rid="Ch1.F4"/>a and c); 168 out of 170 participants rated both avalanches. Of those 168, 78 % indicated the same size, and 15 % rated the avalanche one size larger in the close-up view in Fig. <xref ref-type="fig" rid="Ch1.F4"/>c than in the overview in Fig. <xref ref-type="fig" rid="Ch1.F4"/>a, whereas 7 % rated the avalanche one size smaller in the close-up view compared to the overview. The shift in the proportions is statistically significant (proportion test: <inline-formula><mml:math id="M80" display="inline"><mml:mrow><mml:mi>p</mml:mi><mml:mo>=</mml:mo><mml:mn mathvariant="normal">0.036</mml:mn></mml:mrow></mml:math></inline-formula>).</p>
      <?pagebreak page2903?><p id="d1e2074">When <italic>comparing the results from Europe and North America</italic>, we found the agreement of individual raters in the most frequently estimated size <inline-formula><mml:math id="M81" display="inline"><mml:mrow><mml:msub><mml:mi>s</mml:mi><mml:mi mathvariant="normal">mode</mml:mi></mml:msub></mml:mrow></mml:math></inline-formula> to be identical (<inline-formula><mml:math id="M82" display="inline"><mml:mrow><mml:msub><mml:mi>P</mml:mi><mml:mi mathvariant="normal">mode</mml:mi></mml:msub><mml:mo>=</mml:mo><mml:mn mathvariant="normal">0.66</mml:mn></mml:mrow></mml:math></inline-formula>). This approach slightly favors European respondents, as these contributed a larger share of responses (Europe: <inline-formula><mml:math id="M83" display="inline"><mml:mrow><mml:mi>N</mml:mi><mml:mo>=</mml:mo><mml:mn mathvariant="normal">105</mml:mn></mml:mrow></mml:math></inline-formula>, North America: <inline-formula><mml:math id="M84" display="inline"><mml:mrow><mml:mi>N</mml:mi><mml:mo>=</mml:mo><mml:mn mathvariant="normal">65</mml:mn></mml:mrow></mml:math></inline-formula>). Considering <inline-formula><mml:math id="M85" display="inline"><mml:mrow><mml:msub><mml:mover accent="true"><mml:mi>s</mml:mi><mml:mo mathvariant="normal">‾</mml:mo></mml:mover><mml:mi>j</mml:mi></mml:msub></mml:mrow></mml:math></inline-formula> instead, the agreement <inline-formula><mml:math id="M86" display="inline"><mml:mrow><mml:msub><mml:mi>P</mml:mi><mml:mi mathvariant="normal">correct</mml:mi></mml:msub></mml:mrow></mml:math></inline-formula> was 0.74 overall and 0.66 for both Europe and North America individually. The higher overall agreement results from the definition of the reference size <inline-formula><mml:math id="M87" display="inline"><mml:mrow><mml:msub><mml:mover accent="true"><mml:mi>s</mml:mi><mml:mo mathvariant="normal">‾</mml:mo></mml:mover><mml:mi>j</mml:mi></mml:msub></mml:mrow></mml:math></inline-formula>, where two sizes were considered correct if the reference size was located in between two values. North Americans had a tendency to assign smaller sizes than their European counterparts. This is most notable for the three largest avalanches (avalanches in Fig. <xref ref-type="fig" rid="Ch1.F5"/>b, g and i), with a median size of 4 by Europeans and a median size of 3 by North Americans. With the exception of Fig. <xref ref-type="fig" rid="Ch1.F5"/>e and f, differences in size estimates were statistically significant (Wilcoxon rank-sum test: <inline-formula><mml:math id="M88" display="inline"><mml:mi>p</mml:mi></mml:math></inline-formula> ranging from  <inline-formula><mml:math id="M89" display="inline"><mml:mn mathvariant="normal">0.045</mml:mn></mml:math></inline-formula> to <inline-formula><mml:math id="M90" display="inline"><mml:mrow><mml:mo>&lt;</mml:mo><mml:mn mathvariant="normal">0.001</mml:mn></mml:mrow></mml:math></inline-formula>). Within their continents, respondents had a similar agreement with each other (proportion test: <inline-formula><mml:math id="M91" display="inline"><mml:mrow><mml:mi>p</mml:mi><mml:mo>=</mml:mo><mml:mn mathvariant="normal">1</mml:mn></mml:mrow></mml:math></inline-formula>): on average, <inline-formula><mml:math id="M92" display="inline"><mml:mrow><mml:msub><mml:mi>P</mml:mi><mml:mi mathvariant="normal">agree</mml:mi></mml:msub></mml:mrow></mml:math></inline-formula> was 0.53 within Europe and 0.56 within North America, resulting in rel of 0.73 and 0.75, respectively. Intermediate sizes, which are more commonly used in North America, were chosen in 31 % of the cases by North Americans compared to 23 % by Europeans (proportion test: <inline-formula><mml:math id="M93" display="inline"><mml:mrow><mml:mi>p</mml:mi><mml:mo>=</mml:mo><mml:mn mathvariant="normal">0.368</mml:mn></mml:mrow></mml:math></inline-formula>). When using intermediate sizes, the agreement with the mode intermediate size <inline-formula><mml:math id="M94" display="inline"><mml:mrow><mml:msub><mml:mi>s</mml:mi><mml:mrow><mml:mi mathvariant="normal">mode</mml:mi><mml:mo>.</mml:mo><mml:mi mathvariant="normal">intermediate</mml:mi></mml:mrow></mml:msub></mml:mrow></mml:math></inline-formula> was 53 % for Europe (<inline-formula><mml:math id="M95" display="inline"><mml:mrow><mml:mo>±</mml:mo><mml:mn mathvariant="normal">0.5</mml:mn></mml:mrow></mml:math></inline-formula>: 75 %) and 49% for North America (<inline-formula><mml:math id="M96" display="inline"><mml:mrow><mml:mo>±</mml:mo><mml:mn mathvariant="normal">0.5</mml:mn></mml:mrow></mml:math></inline-formula>: 81 %).</p>

      <?xmltex \floatpos{t}?><fig id="Ch1.F5" specific-use="star"><?xmltex \currentcnt{3}?><?xmltex \def\figurename{Figure}?><label>Figure 3</label><caption><p id="d1e2274">Boxplots showing the size distributions for the 10 avalanches for Europe and North America. Mean values are indicated with <inline-formula><mml:math id="M97" display="inline"><mml:mo>+</mml:mo></mml:math></inline-formula>. Avalanches are labeled according to Fig. <xref ref-type="fig" rid="Ch1.F4"/>, and <bold>(a)</bold> and <bold>(c)</bold> depict the same avalanche. The results from the Wilcoxon rank-sum test indicate that the differences in avalanche size estimation between Europe and North America are significant for 8 out of the 10 avalanches (all except <bold>e</bold> and <bold>f</bold>; <inline-formula><mml:math id="M98" display="inline"><mml:msup><mml:mi/><mml:mo>*</mml:mo></mml:msup></mml:math></inline-formula> (0.01, 0.05], <inline-formula><mml:math id="M99" display="inline"><mml:msup><mml:mi/><mml:mrow><mml:mo>*</mml:mo><mml:mo>*</mml:mo></mml:mrow></mml:msup></mml:math></inline-formula> (0.001, 0.01], <inline-formula><mml:math id="M100" display="inline"><mml:msup><mml:mi/><mml:mrow><mml:mo>*</mml:mo><mml:mo>*</mml:mo><mml:mo>*</mml:mo></mml:mrow></mml:msup></mml:math></inline-formula> <inline-formula><mml:math id="M101" display="inline"><mml:mo>≤</mml:mo></mml:math></inline-formula> 0.001).</p></caption>
          <?xmltex \igopts{width=398.338583pt}?><graphic xlink:href="https://nhess.copernicus.org/articles/23/2895/2023/nhess-23-2895-2023-f03.png"/>

        </fig>

      <p id="d1e2347">Among the <italic>factors used to determine avalanche size</italic> (Table <xref ref-type="table" rid="Ch1.T1"/>), runout was considered the most important with 56 % of respondents considering this factor as very important, followed by volume (very important: 39 %), dimensions (29 %) and destructive potential (20 %) (Fig. <xref ref-type="fig" rid="Ch1.F6"/>). Comparing responses from Europe and North America, we found the most frequent response to be identical for all four factors (very important for runout, important for the other three). However, runout was considered significantly less often as very important in Europe (46 %) compared to North America (72 %, proportion test: <inline-formula><mml:math id="M102" display="inline"><mml:mrow><mml:mi>p</mml:mi><mml:mo>=</mml:mo><mml:mn mathvariant="normal">0.001</mml:mn></mml:mrow></mml:math></inline-formula>).
The factor volume showed a similar pattern with significantly more votes from North America (46 %) than Europe (18 %) for being very important (proportion test: <inline-formula><mml:math id="M103" display="inline"><mml:mrow><mml:mi>p</mml:mi><mml:mo>&lt;</mml:mo><mml:mn mathvariant="normal">0.001</mml:mn></mml:mrow></mml:math></inline-formula>) and the opposite pattern for volume being either less important or not at all important (Europe: 39 %, North America: 7 %, proportion test: <inline-formula><mml:math id="M104" display="inline"><mml:mrow><mml:mi>p</mml:mi><mml:mo>&lt;</mml:mo><mml:mn mathvariant="normal">0.001</mml:mn></mml:mrow></mml:math></inline-formula>). The differences between the continents for rating the importance of destructive potential and dimensions were not significant (proportion test with rating of very important: <inline-formula><mml:math id="M105" display="inline"><mml:mrow><mml:mi>p</mml:mi><mml:mo>=</mml:mo><mml:mn mathvariant="normal">0.260</mml:mn></mml:mrow></mml:math></inline-formula> and <inline-formula><mml:math id="M106" display="inline"><mml:mrow><mml:mi>p</mml:mi><mml:mo>=</mml:mo><mml:mn mathvariant="normal">0.718</mml:mn></mml:mrow></mml:math></inline-formula>).</p>

      <?xmltex \floatpos{t}?><fig id="Ch1.F6" specific-use="star"><?xmltex \currentcnt{4}?><?xmltex \def\figurename{Figure}?><label>Figure 4</label><caption><p id="d1e2420">Comparison of the importance ranking for the factors determining avalanche size for <bold>(a)</bold> Europe and <bold>(b)</bold> North America.</p></caption>
          <?xmltex \igopts{width=398.338583pt}?><graphic xlink:href="https://nhess.copernicus.org/articles/23/2895/2023/nhess-23-2895-2023-f04.png"/>

        </fig>

</sec>
<sec id="Ch1.S4.SS2">
  <label>4.2</label><title>Avalanche mapping from oblique photographs (study 2)</title>
      <p id="d1e2443">The 10 participants, all very familiar with the study area, centered the avalanches based on oblique photographs around the corresponding reference mapping (Fig. <xref ref-type="fig" rid="Ch1.F7"/>), identifying the correct locations.</p>

      <?xmltex \floatpos{t}?><fig id="Ch1.F7" specific-use="star"><?xmltex \currentcnt{5}?><?xmltex \def\figurename{Figure}?><label>Figure 5</label><caption><p id="d1e2450">Heat map illustrating expert agreement on avalanche area for the six avalanches mapped from oblique photographs. Dark blue indicates areas of very good agreement, identified as part of an avalanche by all 10 experts. For location and size comparison the outlines of the avalanches, mapped from the photographs georeferenced with the monoplotting tool by <xref ref-type="bibr" rid="bib1.bibx6 bib1.bibx7" id="text.52"/>, are shown as a reference in pink (for avalanche <bold>a</bold> the lower part was occluded by a tree; map source: Federal Office of Topography).</p></caption>
          <?xmltex \igopts{width=426.791339pt}?><graphic xlink:href="https://nhess.copernicus.org/articles/23/2895/2023/nhess-23-2895-2023-f05.jpg"/>

        </fig>

      <p id="d1e2465">On average, the overlapping proportion of the mappings of any two participants of IoU<inline-formula><mml:math id="M107" display="inline"><mml:msub><mml:mi/><mml:mi mathvariant="normal">pairwise</mml:mi></mml:msub></mml:math></inline-formula> was 0.52, varying from 0.32 for the worst pairwise agreement to 0.69 for the best one (Table <xref ref-type="table" rid="Ch1.T3"/>). Individual pairwise comparisons are shown in Fig. <xref ref-type="fig" rid="App1.Ch1.S1.F11"/> in the Appendix. When comparing individual mappings to the area mapped by at least one person as AoU, the mean IoU<inline-formula><mml:math id="M108" display="inline"><mml:msub><mml:mi/><mml:mi mathvariant="normal">all</mml:mi></mml:msub></mml:math></inline-formula> is 0.31, ranging from 0.21 to 0.41. Only a fraction of 9 % of the combined area of union (AoU for all 10 participants) was identified by all participants as an avalanche (area of overlap for all 10 participants, AoO) showing the considerable scatter of individual mappings. Comparing individual mappings to the reference resulted in a mean IoU<inline-formula><mml:math id="M109" display="inline"><mml:msub><mml:mi/><mml:mi mathvariant="normal">ref<?pagebreak page2904?></mml:mi></mml:msub></mml:math></inline-formula> of 0.60, with a minimum of 0.40 and a maximum of 0.80. These areas of higher agreement between participants, visible in darker hues in Fig. <xref ref-type="fig" rid="Ch1.F7"/>, coincide with the outlines from the reference mapping, highlighting that variations happened around the reference. In other words, individual mappings had a higher correspondence with the reference mapping compared to mappings by other individuals (see also Fig. <xref ref-type="fig" rid="App1.Ch1.S1.F11"/> in the Appendix). The large variation between individual mappings was also shown when analyzing the absolute values of the mapped areas (Table <xref ref-type="table" rid="Ch1.T4"/>): the largest mapped area was between 2 and 4 times larger than the smallest mapped area (avalanches in Fig. <xref ref-type="fig" rid="Ch1.F7"/>b and f). Additionally, the comparison with the reference showed a systematic tendency towards underestimation of the area, as in all cases the median mapped area was between 10 % and 36 % smaller than the reference area.</p>

<?xmltex \floatpos{t}?><table-wrap id="Ch1.T3"><?xmltex \currentcnt{3}?><label>Table 3</label><caption><p id="d1e2512">Intersection over union (IoU) for avalanches mapped from oblique photographs (study 2). Values represent the mean of six avalanches.</p></caption><oasis:table frame="topbot"><oasis:tgroup cols="4">
     <oasis:colspec colnum="1" colname="col1" align="left"/>
     <oasis:colspec colnum="2" colname="col2" align="right"/>
     <oasis:colspec colnum="3" colname="col3" align="right"/>
     <oasis:colspec colnum="4" colname="col4" align="right"/>
     <oasis:thead>
       <oasis:row rowsep="1">
         <oasis:entry colname="col1"/>
         <oasis:entry colname="col2">IoU<inline-formula><mml:math id="M111" display="inline"><mml:msub><mml:mi/><mml:mi mathvariant="normal">pairwise</mml:mi></mml:msub></mml:math></inline-formula></oasis:entry>
         <oasis:entry colname="col3">IoU<inline-formula><mml:math id="M112" display="inline"><mml:msub><mml:mi/><mml:mi mathvariant="normal">all</mml:mi></mml:msub></mml:math></inline-formula></oasis:entry>
         <oasis:entry colname="col4">IoU<inline-formula><mml:math id="M113" display="inline"><mml:mrow><mml:msubsup><mml:mi/><mml:mi mathvariant="normal">ref</mml:mi><mml:mo>*</mml:mo></mml:msubsup></mml:mrow></mml:math></inline-formula></oasis:entry>
       </oasis:row>
     </oasis:thead>
     <oasis:tbody>
       <oasis:row>
         <oasis:entry colname="col1">Mean</oasis:entry>
         <oasis:entry colname="col2">0.52</oasis:entry>
         <oasis:entry colname="col3">0.31</oasis:entry>
         <oasis:entry colname="col4">0.60</oasis:entry>
       </oasis:row>
       <oasis:row>
         <oasis:entry colname="col1">Min</oasis:entry>
         <oasis:entry colname="col2">0.32</oasis:entry>
         <oasis:entry colname="col3">0.21</oasis:entry>
         <oasis:entry colname="col4">0.40</oasis:entry>
       </oasis:row>
       <oasis:row>
         <oasis:entry colname="col1">Max</oasis:entry>
         <oasis:entry colname="col2">0.69</oasis:entry>
         <oasis:entry colname="col3">0.41</oasis:entry>
         <oasis:entry colname="col4">0.80</oasis:entry>
       </oasis:row>
     </oasis:tbody>
   </oasis:tgroup></oasis:table><table-wrap-foot><p id="d1e2515"><inline-formula><mml:math id="M110" display="inline"><mml:msup><mml:mi/><mml:mo>*</mml:mo></mml:msup></mml:math></inline-formula> Without deposit from Fig. <xref ref-type="fig" rid="Ch1.F7"/>a.</p></table-wrap-foot><?xmltex \gdef\@currentlabel{3}?></table-wrap>

<?xmltex \floatpos{t}?><table-wrap id="Ch1.T4" specific-use="star"><?xmltex \currentcnt{4}?><label>Table 4</label><caption><p id="d1e2637">Avalanches mapped from oblique photographs (numbering corresponds to Fig. <xref ref-type="fig" rid="Ch1.F7"/>, study 2). Shown are the areas of the reference mapping and the respective median, minimum and maximum of the 10 individual mappings. The relative difference to the reference (in %) is indicated in parentheses.</p></caption><oasis:table frame="topbot"><oasis:tgroup cols="8">
     <oasis:colspec colnum="1" colname="col1" align="left"/>
     <oasis:colspec colnum="2" colname="col2" align="right"/>
     <oasis:colspec colnum="3" colname="col3" align="right"/>
     <oasis:colspec colnum="4" colname="col4" align="right" colsep="1"/>
     <oasis:colspec colnum="5" colname="col5" align="right"/>
     <oasis:colspec colnum="6" colname="col6" align="right" colsep="1"/>
     <oasis:colspec colnum="7" colname="col7" align="right"/>
     <oasis:colspec colnum="8" colname="col8" align="right"/>
     <oasis:thead>
       <oasis:row>
         <oasis:entry colname="col1">Avalanche</oasis:entry>
         <oasis:entry colname="col2">Reference</oasis:entry>
         <oasis:entry namest="col3" nameend="col4" align="center" colsep="1">Median (m<inline-formula><mml:math id="M115" display="inline"><mml:msup><mml:mi/><mml:mn mathvariant="normal">2</mml:mn></mml:msup></mml:math></inline-formula>) </oasis:entry>
         <oasis:entry namest="col5" nameend="col6" align="center" colsep="1">Min (m<inline-formula><mml:math id="M116" display="inline"><mml:msup><mml:mi/><mml:mn mathvariant="normal">2</mml:mn></mml:msup></mml:math></inline-formula>) </oasis:entry>
         <oasis:entry namest="col7" nameend="col8" align="center">Max (m<inline-formula><mml:math id="M117" display="inline"><mml:msup><mml:mi/><mml:mn mathvariant="normal">2</mml:mn></mml:msup></mml:math></inline-formula>) </oasis:entry>
       </oasis:row>
       <oasis:row rowsep="1">
         <oasis:entry colname="col1"/>
         <oasis:entry colname="col2">(m<inline-formula><mml:math id="M118" display="inline"><mml:msup><mml:mi/><mml:mn mathvariant="normal">2</mml:mn></mml:msup></mml:math></inline-formula>)</oasis:entry>
         <oasis:entry colname="col3"/>
         <oasis:entry colname="col4"/>
         <oasis:entry colname="col5"/>
         <oasis:entry colname="col6"/>
         <oasis:entry colname="col7"/>
         <oasis:entry colname="col8"/>
       </oasis:row>
     </oasis:thead>
     <oasis:tbody>
       <oasis:row>
         <oasis:entry colname="col1">(a)<inline-formula><mml:math id="M119" display="inline"><mml:msup><mml:mi/><mml:mo>*</mml:mo></mml:msup></mml:math></inline-formula></oasis:entry>
         <oasis:entry colname="col2">118 615</oasis:entry>
         <oasis:entry colname="col3">106 566</oasis:entry>
         <oasis:entry colname="col4">(<inline-formula><mml:math id="M120" display="inline"><mml:mo lspace="0mm">-</mml:mo></mml:math></inline-formula>10)</oasis:entry>
         <oasis:entry colname="col5">66 143</oasis:entry>
         <oasis:entry colname="col6">(<inline-formula><mml:math id="M121" display="inline"><mml:mo lspace="0mm">-</mml:mo></mml:math></inline-formula>44)</oasis:entry>
         <oasis:entry colname="col7">190 823</oasis:entry>
         <oasis:entry colname="col8">(<inline-formula><mml:math id="M122" display="inline"><mml:mo lspace="0mm">+</mml:mo></mml:math></inline-formula>61)</oasis:entry>
       </oasis:row>
       <oasis:row>
         <oasis:entry colname="col1">(b)</oasis:entry>
         <oasis:entry colname="col2">13 673</oasis:entry>
         <oasis:entry colname="col3">8741</oasis:entry>
         <oasis:entry colname="col4">(<inline-formula><mml:math id="M123" display="inline"><mml:mo lspace="0mm">-</mml:mo></mml:math></inline-formula>36)</oasis:entry>
         <oasis:entry colname="col5">3745</oasis:entry>
         <oasis:entry colname="col6">(<inline-formula><mml:math id="M124" display="inline"><mml:mo lspace="0mm">-</mml:mo></mml:math></inline-formula>73)</oasis:entry>
         <oasis:entry colname="col7">15 518</oasis:entry>
         <oasis:entry colname="col8">(<inline-formula><mml:math id="M125" display="inline"><mml:mo lspace="0mm">+</mml:mo></mml:math></inline-formula>14)</oasis:entry>
       </oasis:row>
       <oasis:row>
         <oasis:entry colname="col1">(c)</oasis:entry>
         <oasis:entry colname="col2">13 082</oasis:entry>
         <oasis:entry colname="col3">11 071</oasis:entry>
         <oasis:entry colname="col4">(<inline-formula><mml:math id="M126" display="inline"><mml:mo lspace="0mm">-</mml:mo></mml:math></inline-formula>15)</oasis:entry>
         <oasis:entry colname="col5">7422</oasis:entry>
         <oasis:entry colname="col6">(<inline-formula><mml:math id="M127" display="inline"><mml:mo lspace="0mm">-</mml:mo></mml:math></inline-formula>43)</oasis:entry>
         <oasis:entry colname="col7">16 221</oasis:entry>
         <oasis:entry colname="col8">(<inline-formula><mml:math id="M128" display="inline"><mml:mo lspace="0mm">+</mml:mo></mml:math></inline-formula>24)</oasis:entry>
       </oasis:row>
       <oasis:row>
         <oasis:entry colname="col1">(d)</oasis:entry>
         <oasis:entry colname="col2">6570</oasis:entry>
         <oasis:entry colname="col3">5422</oasis:entry>
         <oasis:entry colname="col4">(<inline-formula><mml:math id="M129" display="inline"><mml:mo lspace="0mm">-</mml:mo></mml:math></inline-formula>18)</oasis:entry>
         <oasis:entry colname="col5">2183</oasis:entry>
         <oasis:entry colname="col6">(<inline-formula><mml:math id="M130" display="inline"><mml:mo lspace="0mm">-</mml:mo></mml:math></inline-formula>67)</oasis:entry>
         <oasis:entry colname="col7">7533</oasis:entry>
         <oasis:entry colname="col8">(<inline-formula><mml:math id="M131" display="inline"><mml:mo lspace="0mm">+</mml:mo></mml:math></inline-formula>15)</oasis:entry>
       </oasis:row>
       <oasis:row>
         <oasis:entry colname="col1">(e)</oasis:entry>
         <oasis:entry colname="col2">63 807</oasis:entry>
         <oasis:entry colname="col3">50 127</oasis:entry>
         <oasis:entry colname="col4">(<inline-formula><mml:math id="M132" display="inline"><mml:mo lspace="0mm">-</mml:mo></mml:math></inline-formula>21)</oasis:entry>
         <oasis:entry colname="col5">24 804</oasis:entry>
         <oasis:entry colname="col6">(<inline-formula><mml:math id="M133" display="inline"><mml:mo lspace="0mm">-</mml:mo></mml:math></inline-formula>61)</oasis:entry>
         <oasis:entry colname="col7">64 680</oasis:entry>
         <oasis:entry colname="col8">(<inline-formula><mml:math id="M134" display="inline"><mml:mo lspace="0mm">+</mml:mo></mml:math></inline-formula>1)</oasis:entry>
       </oasis:row>
       <oasis:row>
         <oasis:entry colname="col1">(f)</oasis:entry>
         <oasis:entry colname="col2">90 400</oasis:entry>
         <oasis:entry colname="col3">71 967</oasis:entry>
         <oasis:entry colname="col4">(<inline-formula><mml:math id="M135" display="inline"><mml:mo lspace="0mm">-</mml:mo></mml:math></inline-formula>20)</oasis:entry>
         <oasis:entry colname="col5">58 383</oasis:entry>
         <oasis:entry colname="col6">(<inline-formula><mml:math id="M136" display="inline"><mml:mo lspace="0mm">-</mml:mo></mml:math></inline-formula>35)</oasis:entry>
         <oasis:entry colname="col7">114 404</oasis:entry>
         <oasis:entry colname="col8">(<inline-formula><mml:math id="M137" display="inline"><mml:mo lspace="0mm">+</mml:mo></mml:math></inline-formula>23)</oasis:entry>
       </oasis:row>
     </oasis:tbody>
   </oasis:tgroup></oasis:table><table-wrap-foot><p id="d1e2642"><inline-formula><mml:math id="M114" display="inline"><mml:msup><mml:mi/><mml:mo>*</mml:mo></mml:msup></mml:math></inline-formula> Without the lower part of deposit.</p></table-wrap-foot><?xmltex \gdef\@currentlabel{4}?></table-wrap>

      <?pagebreak page2905?><p id="d1e3059">Two of the raters had statistically lower pairwise overlap in their area of avalanche activity (mean IoU<inline-formula><mml:math id="M138" display="inline"><mml:mrow><mml:msub><mml:mi/><mml:mi mathvariant="normal">pairwise</mml:mi></mml:msub><mml:mo>≤</mml:mo></mml:mrow></mml:math></inline-formula> 0.49) compared to the other eight raters (mean IoU<inline-formula><mml:math id="M139" display="inline"><mml:mrow><mml:msub><mml:mi/><mml:mi mathvariant="normal">pairwise</mml:mi></mml:msub><mml:mo>≥</mml:mo></mml:mrow></mml:math></inline-formula> 0.69, Wilcoxon rank-sum test: <inline-formula><mml:math id="M140" display="inline"><mml:mrow><mml:mi>p</mml:mi><mml:mo>=</mml:mo><mml:mn mathvariant="normal">0.011</mml:mn></mml:mrow></mml:math></inline-formula> and <inline-formula><mml:math id="M141" display="inline"><mml:mrow><mml:mi>p</mml:mi><mml:mo>=</mml:mo><mml:mn mathvariant="normal">0.002</mml:mn></mml:mrow></mml:math></inline-formula>; see Fig. <xref ref-type="fig" rid="App1.Ch1.S1.F11"/> in the Appendix). These two raters also had the lowest agreement with the reference mapping (IoU<inline-formula><mml:math id="M142" display="inline"><mml:mrow><mml:msub><mml:mi/><mml:mi mathvariant="normal">ref</mml:mi></mml:msub><mml:mo>≤</mml:mo></mml:mrow></mml:math></inline-formula> 0.44), lower than the other eight (IoU<inline-formula><mml:math id="M143" display="inline"><mml:mrow><mml:msub><mml:mi/><mml:mi mathvariant="normal">ref</mml:mi></mml:msub><mml:mo>≥</mml:mo></mml:mrow></mml:math></inline-formula> 0.53).</p>
</sec>
<sec id="Ch1.S4.SS3">
  <label>4.3</label><title>Avalanche mapping from remotely sensed imagery (study 3)</title>
      <p id="d1e3145">When visually comparing the mappings, differences can be observed not only between image resolutions (Fig. <xref ref-type="fig" rid="Ch1.F8"/>) but also between participants (Fig. <xref ref-type="fig" rid="Ch1.F9"/>). The mean of the pairwise overlapping proportion of avalanches of IoU<inline-formula><mml:math id="M144" display="inline"><mml:msub><mml:mi/><mml:mi mathvariant="normal">pairwise</mml:mi></mml:msub></mml:math></inline-formula> increased with increasing image resolution from 0.46 at 2 m resolution to 0.68 at 25 <inline-formula><mml:math id="M145" display="inline"><mml:mrow class="unit"><mml:mi mathvariant="normal">cm</mml:mi></mml:mrow></mml:math></inline-formula> resolution (Table <xref ref-type="table" rid="Ch1.T5"/>). Considering the area classified as an avalanche by three or more raters as the reference showed an increase in IoU<inline-formula><mml:math id="M146" display="inline"><mml:msub><mml:mi/><mml:mi mathvariant="normal">ref</mml:mi></mml:msub></mml:math></inline-formula> from 0.64 at 2 m resolution to 0.80 at 25 <inline-formula><mml:math id="M147" display="inline"><mml:mrow class="unit"><mml:mi mathvariant="normal">cm</mml:mi></mml:mrow></mml:math></inline-formula> resolution (Table <xref ref-type="table" rid="Ch1.T6"/>). Regarding the influence of illumination conditions, all IoU<inline-formula><mml:math id="M148" display="inline"><mml:msub><mml:mi/><mml:mi mathvariant="normal">pairwise</mml:mi></mml:msub></mml:math></inline-formula> scores were higher in illuminated areas compared to shaded areas of the image (for instance, at 25 <inline-formula><mml:math id="M149" display="inline"><mml:mrow class="unit"><mml:mi mathvariant="normal">cm</mml:mi></mml:mrow></mml:math></inline-formula> resolution – illuminated: 0.77, shaded: 0.54; Table <xref ref-type="table" rid="Ch1.T5"/>). Snow conditions also influenced the agreement of the mappings (Figs. <xref ref-type="fig" rid="Ch1.F8"/> and <xref ref-type="fig" rid="Ch1.F9"/>): the mean IoU<inline-formula><mml:math id="M150" display="inline"><mml:msub><mml:mi/><mml:mi mathvariant="normal">pairwise</mml:mi></mml:msub></mml:math></inline-formula> was higher in wet-snow conditions (25 <inline-formula><mml:math id="M151" display="inline"><mml:mrow class="unit"><mml:mi mathvariant="normal">cm</mml:mi></mml:mrow></mml:math></inline-formula> resolution: 0.90) compared to dry-snow conditions (25 <inline-formula><mml:math id="M152" display="inline"><mml:mrow class="unit"><mml:mi mathvariant="normal">cm</mml:mi></mml:mrow></mml:math></inline-formula> resolution: 0.66; Table <xref ref-type="table" rid="Ch1.T5"/>). Moreover, individual mappings were also much more similar in wet-snow conditions compared to dry-snow conditions with the variations in IoU<inline-formula><mml:math id="M153" display="inline"><mml:msub><mml:mi/><mml:mi mathvariant="normal">pairwise</mml:mi></mml:msub></mml:math></inline-formula> ranging for dry-snow conditions at 25 <inline-formula><mml:math id="M154" display="inline"><mml:mrow class="unit"><mml:mi mathvariant="normal">cm</mml:mi></mml:mrow></mml:math></inline-formula> resolution between 0.56 and 0.77 (mean: 0.66; standard deviation: 0.07) and for wet-snow conditions between 0.88 and 0.91 (mean: 0.9; standard deviation: 0.01; Fig. <xref ref-type="fig" rid="Ch1.F10"/>d). Overall, variations in mean IoU<inline-formula><mml:math id="M155" display="inline"><mml:msub><mml:mi/><mml:mi mathvariant="normal">pairwise</mml:mi></mml:msub></mml:math></inline-formula> were smaller across resolutions (0.02 to 0.22) than the differences between the minimum and maximum IoU<inline-formula><mml:math id="M156" display="inline"><mml:msub><mml:mi/><mml:mi mathvariant="normal">pairwise</mml:mi></mml:msub></mml:math></inline-formula> within one resolution (0.20 for 25 <inline-formula><mml:math id="M157" display="inline"><mml:mrow class="unit"><mml:mi mathvariant="normal">cm</mml:mi></mml:mrow></mml:math></inline-formula> resolution to 0.43 for 2 m resolution). This is especially pronounced for dry-snow conditions (Fig. <xref ref-type="fig" rid="Ch1.F10"/>). The large variations between different experts are also reflected in the avalanche area that was<?pagebreak page2906?> consistently identified by one person over all four spatial resolutions (dark red in Fig. <xref ref-type="fig" rid="Ch1.F9"/>).</p>

<?xmltex \floatpos{t}?><table-wrap id="Ch1.T5"><?xmltex \currentcnt{5}?><label>Table 5</label><caption><p id="d1e3296">Mean IoU<inline-formula><mml:math id="M158" display="inline"><mml:msub><mml:mi/><mml:mi mathvariant="normal">pairwise</mml:mi></mml:msub></mml:math></inline-formula> for different subsets and spatial resolutions.</p></caption><oasis:table frame="topbot"><oasis:tgroup cols="6">
     <oasis:colspec colnum="1" colname="col1" align="left"/>
     <oasis:colspec colnum="2" colname="col2" align="right"/>
     <oasis:colspec colnum="3" colname="col3" align="right"/>
     <oasis:colspec colnum="4" colname="col4" align="right"/>
     <oasis:colspec colnum="5" colname="col5" align="right"/>
     <oasis:colspec colnum="6" colname="col6" align="right"/>
     <oasis:thead>
       <oasis:row>
         <oasis:entry colname="col1"/>
         <oasis:entry rowsep="1" namest="col2" nameend="col5" align="center">Image resolution </oasis:entry>
         <oasis:entry colname="col6">Area</oasis:entry>
       </oasis:row>
       <oasis:row rowsep="1">
         <oasis:entry colname="col1">Subset</oasis:entry>
         <oasis:entry colname="col2">2 m</oasis:entry>
         <oasis:entry colname="col3">1 m</oasis:entry>
         <oasis:entry colname="col4">50 <inline-formula><mml:math id="M159" display="inline"><mml:mrow class="unit"><mml:mi mathvariant="normal">cm</mml:mi></mml:mrow></mml:math></inline-formula></oasis:entry>
         <oasis:entry colname="col5">25 <inline-formula><mml:math id="M160" display="inline"><mml:mrow class="unit"><mml:mi mathvariant="normal">cm</mml:mi></mml:mrow></mml:math></inline-formula></oasis:entry>
         <oasis:entry colname="col6">(km<inline-formula><mml:math id="M161" display="inline"><mml:msup><mml:mi/><mml:mn mathvariant="normal">2</mml:mn></mml:msup></mml:math></inline-formula>)</oasis:entry>
       </oasis:row>
     </oasis:thead>
     <oasis:tbody>
       <oasis:row rowsep="1">
         <oasis:entry colname="col1">Overall</oasis:entry>
         <oasis:entry colname="col2">0.46</oasis:entry>
         <oasis:entry colname="col3">0.61</oasis:entry>
         <oasis:entry colname="col4">0.66</oasis:entry>
         <oasis:entry colname="col5">0.68</oasis:entry>
         <oasis:entry colname="col6">2.9</oasis:entry>
       </oasis:row>
       <oasis:row>
         <oasis:entry colname="col1">Illuminated</oasis:entry>
         <oasis:entry colname="col2">0.51</oasis:entry>
         <oasis:entry colname="col3">0.67</oasis:entry>
         <oasis:entry colname="col4">0.73</oasis:entry>
         <oasis:entry colname="col5">0.77</oasis:entry>
         <oasis:entry colname="col6">1.6</oasis:entry>
       </oasis:row>
       <oasis:row>
         <oasis:entry colname="col1">Shaded</oasis:entry>
         <oasis:entry colname="col2">0.36</oasis:entry>
         <oasis:entry colname="col3">0.49</oasis:entry>
         <oasis:entry colname="col4">0.54</oasis:entry>
         <oasis:entry colname="col5">0.54</oasis:entry>
         <oasis:entry colname="col6">1.3</oasis:entry>
       </oasis:row>
       <oasis:row>
         <oasis:entry colname="col1">Dry snow</oasis:entry>
         <oasis:entry colname="col2">0.44</oasis:entry>
         <oasis:entry colname="col3">0.59</oasis:entry>
         <oasis:entry colname="col4">0.64</oasis:entry>
         <oasis:entry colname="col5">0.66</oasis:entry>
         <oasis:entry colname="col6">2.2</oasis:entry>
       </oasis:row>
       <oasis:row>
         <oasis:entry colname="col1">Wet snow</oasis:entry>
         <oasis:entry colname="col2">0.70</oasis:entry>
         <oasis:entry colname="col3">0.81</oasis:entry>
         <oasis:entry colname="col4">0.86</oasis:entry>
         <oasis:entry colname="col5">0.90</oasis:entry>
         <oasis:entry colname="col6">0.7</oasis:entry>
       </oasis:row>
     </oasis:tbody>
   </oasis:tgroup></oasis:table><?xmltex \gdef\@currentlabel{5}?></table-wrap>

      <?xmltex \floatpos{t}?><fig id="Ch1.F8"><?xmltex \currentcnt{6}?><?xmltex \def\figurename{Figure}?><label>Figure 6</label><caption><p id="d1e3500">Heat map illustrating expert agreement on the avalanche area mapped from remotely sensed imagery for four spatial resolutions (2 m to 25 <inline-formula><mml:math id="M162" display="inline"><mml:mrow class="unit"><mml:mi mathvariant="normal">cm</mml:mi></mml:mrow></mml:math></inline-formula>, rows, from top to bottom) for the example dry-snow conditions (left column) and wet-snow conditions (right column). The darker the hue, the greater the agreement of the five experts on the existence of an avalanche in that particular location (map source: Federal Office of Topography).</p></caption>
          <?xmltex \igopts{width=236.157874pt}?><graphic xlink:href="https://nhess.copernicus.org/articles/23/2895/2023/nhess-23-2895-2023-f06.jpg"/>

          <p id="d1e3516">.</p>
        </fig>

      <?xmltex \floatpos{t}?><fig id="Ch1.F9"><?xmltex \currentcnt{7}?><?xmltex \def\figurename{Figure}?><label>Figure 7</label><caption><p id="d1e3523">Heat map showing differences in the avalanche mappings for participants A to E (rows, from top to bottom), as a function of the four resolutions for the example dry-snow conditions (left column) and wet-snow conditions (right column). Dark hues (red) indicate areas where an avalanche was detected in all four resolutions; light hues indicate areas where an avalanche was detected in only one resolution (map source: Federal Office of Topography).</p></caption>
          <?xmltex \igopts{width=236.157874pt}?><graphic xlink:href="https://nhess.copernicus.org/articles/23/2895/2023/nhess-23-2895-2023-f07.jpg"/>

        </fig>

      <?xmltex \floatpos{t}?><fig id="Ch1.F10" specific-use="star"><?xmltex \currentcnt{8}?><?xmltex \def\figurename{Figure}?><label>Figure 8</label><caption><p id="d1e3534">IoU<inline-formula><mml:math id="M163" display="inline"><mml:msub><mml:mi/><mml:mi mathvariant="normal">pairwise</mml:mi></mml:msub></mml:math></inline-formula> for dry- (below diagonal) and wet-snow conditions (above diagonal). The letters A to E represent the different participants; the four tiles <bold>(a–d)</bold> represent the four resolutions.</p></caption>
          <?xmltex \igopts{width=398.338583pt}?><graphic xlink:href="https://nhess.copernicus.org/articles/23/2895/2023/nhess-23-2895-2023-f08.png"/>

        </fig>

      <p id="d1e3555">One of the five participants had a lower pairwise agreement compared to the other four (for instance at 2 m resolution: IoU<inline-formula><mml:math id="M164" display="inline"><mml:mrow><mml:msub><mml:mi/><mml:mi mathvariant="normal">pairwise</mml:mi></mml:msub><mml:mo>≤</mml:mo></mml:mrow></mml:math></inline-formula> 0.41 vs. IoU<inline-formula><mml:math id="M165" display="inline"><mml:msub><mml:mi/><mml:mi mathvariant="normal">pairwise</mml:mi></mml:msub></mml:math></inline-formula> of 0.37–0.76), although this was not significant (Wilcoxon rank-sum test: 2 m resolution of <inline-formula><mml:math id="M166" display="inline"><mml:mrow><mml:mi>p</mml:mi><mml:mo>=</mml:mo><mml:mn mathvariant="normal">0.088</mml:mn></mml:mrow></mml:math></inline-formula>, 25 <inline-formula><mml:math id="M167" display="inline"><mml:mrow class="unit"><mml:mi mathvariant="normal">cm</mml:mi></mml:mrow></mml:math></inline-formula> resolution of <inline-formula><mml:math id="M168" display="inline"><mml:mrow><mml:mi>p</mml:mi><mml:mo>=</mml:mo><mml:mn mathvariant="normal">0.055</mml:mn></mml:mrow></mml:math></inline-formula>).
Considering the area classified as avalanche by three or more raters as the best approximation of a ground truth, the mean agreement with this mapping ranged between IoU<inline-formula><mml:math id="M169" display="inline"><mml:mrow><mml:msub><mml:mi/><mml:mi mathvariant="normal">ref</mml:mi></mml:msub><mml:mo>=</mml:mo><mml:mn mathvariant="normal">0.64</mml:mn></mml:mrow></mml:math></inline-formula> and IoU<inline-formula><mml:math id="M170" display="inline"><mml:mrow><mml:msub><mml:mi/><mml:mi mathvariant="normal">ref</mml:mi></mml:msub><mml:mo>=</mml:mo><mml:mn mathvariant="normal">0.80</mml:mn></mml:mrow></mml:math></inline-formula> (Table <xref ref-type="table" rid="Ch1.T6"/>). Again the same participant had the lowest mean agreement.</p>

<?xmltex \floatpos{t}?><table-wrap id="Ch1.T6"><?xmltex \currentcnt{6}?><label>Table 6</label><caption><p id="d1e3645">Mean IoU<inline-formula><mml:math id="M171" display="inline"><mml:msub><mml:mi/><mml:mi mathvariant="normal">ref</mml:mi></mml:msub></mml:math></inline-formula> for all spatial resolutions.</p></caption><oasis:table frame="topbot"><oasis:tgroup cols="5">
     <oasis:colspec colnum="1" colname="col1" align="left"/>
     <oasis:colspec colnum="2" colname="col2" align="right"/>
     <oasis:colspec colnum="3" colname="col3" align="right"/>
     <oasis:colspec colnum="4" colname="col4" align="right"/>
     <oasis:colspec colnum="5" colname="col5" align="right"/>
     <oasis:thead>
       <oasis:row>
         <oasis:entry colname="col1"/>
         <oasis:entry rowsep="1" namest="col2" nameend="col5" align="center">Image resolution </oasis:entry>
       </oasis:row>
       <oasis:row rowsep="1">
         <oasis:entry colname="col1"/>
         <oasis:entry colname="col2">2 m</oasis:entry>
         <oasis:entry colname="col3">1 m</oasis:entry>
         <oasis:entry colname="col4">50 <inline-formula><mml:math id="M172" display="inline"><mml:mrow class="unit"><mml:mi mathvariant="normal">cm</mml:mi></mml:mrow></mml:math></inline-formula></oasis:entry>
         <oasis:entry colname="col5">25 <inline-formula><mml:math id="M173" display="inline"><mml:mrow class="unit"><mml:mi mathvariant="normal">cm</mml:mi></mml:mrow></mml:math></inline-formula></oasis:entry>
       </oasis:row>
     </oasis:thead>
     <oasis:tbody>
       <oasis:row>
         <oasis:entry colname="col1">IoU<inline-formula><mml:math id="M174" display="inline"><mml:msub><mml:mi/><mml:mi mathvariant="normal">ref</mml:mi></mml:msub></mml:math></inline-formula></oasis:entry>
         <oasis:entry colname="col2">0.64</oasis:entry>
         <oasis:entry colname="col3">0.76</oasis:entry>
         <oasis:entry colname="col4">0.79</oasis:entry>
         <oasis:entry colname="col5">0.80</oasis:entry>
       </oasis:row>
     </oasis:tbody>
   </oasis:tgroup></oasis:table><?xmltex \gdef\@currentlabel{6}?></table-wrap>

</sec>
</sec>
<sec id="Ch1.S5">
  <label>5</label><title>Discussion</title>
      <p id="d1e3754">We explored the reliability of estimates of avalanche size and detecting the outline of avalanches from images. The key findings are as follows.
<list list-type="bullet"><list-item>
      <?pagebreak page2907?><p id="d1e3759">The agreement rate <inline-formula><mml:math id="M175" display="inline"><mml:mrow><mml:msub><mml:mi>P</mml:mi><mml:mi mathvariant="normal">agree</mml:mi></mml:msub></mml:mrow></mml:math></inline-formula> between any two size estimates was 0.53, resulting in a reliability rel of 0.72, while the agreement with the avalanche size considered correct  was 0.74, with the most frequently chosen size of 0.66 (mode).</p></list-item><list-item>
      <p id="d1e3774">Significant differences were observed between Europe and North America, both for rating avalanche size and for weighing the factors determining avalanche size.</p></list-item><list-item>
      <p id="d1e3778">The mean overlapping proportion of any two avalanche mappings of IoU<inline-formula><mml:math id="M176" display="inline"><mml:msub><mml:mi/><mml:mi mathvariant="normal">pairwise</mml:mi></mml:msub></mml:math></inline-formula> was 0.52 (study 2) and between 0.46 and 0.68 (study 3) and thus lower than the mean agreement with the reference of IoU<inline-formula><mml:math id="M177" display="inline"><mml:msub><mml:mi/><mml:mi mathvariant="normal">ref</mml:mi></mml:msub></mml:math></inline-formula>, which was 0.60 (study 2) and between 0.64 and 0.8 (study 3).</p></list-item></list>
In the following, we discuss these results by considering definitions, the conclusiveness of the data for the task at hand and the competence of participants. Finally, we provide recommendations for practice.</p>
<sec id="Ch1.S5.SS1">
  <label>5.1</label><title>Avalanche size estimation</title>
      <p id="d1e3807">Our results show that it is difficult to achieve consistent size estimates of avalanches: in only 53 % of the cases did any two size estimates agree, and in 66 % of the cases an individual estimate agreed with the most frequent size among the respondents (mode). It showed, however, that in most cases disagreements were comparably small with 92 % of individual estimates being either equal to the mode <inline-formula><mml:math id="M178" display="inline"><mml:mrow><mml:msub><mml:mi>s</mml:mi><mml:mi mathvariant="normal">mode</mml:mi></mml:msub></mml:mrow></mml:math></inline-formula> or equal to the second-most frequent neighboring size or, if considering intermediate sizes, that 74 % of the estimates were within one intermediate size class. Comparing our results to previous studies investigating agreement for avalanche size estimates (Table <xref ref-type="table" rid="Ch1.T7"/>), the agreement rate with the most frequent size ranged from 62 % <xref ref-type="bibr" rid="bib1.bibx33" id="paren.53"/> to 84 % <xref ref-type="bibr" rid="bib1.bibx20" id="paren.54"/>. The high agreement rates in <xref ref-type="bibr" rid="bib1.bibx24" id="text.55"/> and <xref ref-type="bibr" rid="bib1.bibx20" id="text.56"/> are probably related to the fact that these studies relied on a small number of experienced practitioners with a comparably similar background and training. In contrast, both the studies by <xref ref-type="bibr" rid="bib1.bibx33" id="text.57"/> and our study included participants from numerous countries and thus respondents with different avalanche backgrounds, leading to a more diverse group of avalanche practitioners. Moreover, <xref ref-type="bibr" rid="bib1.bibx33" id="text.58"/> speculated that the changes introduced in the avalanche size definitions shortly before their survey may have lowered the agreement. The reliability of an individual size estimate in this study was 0.73, highlighting the uncertainty associated with these data. Thus, using size estimates by an individual as ground truth when developing or evaluating models, a perfect model can achieve more than 73 % accuracy only by chance if the errors a model makes are independent from the errors contained in the avalanche size labels.</p>

<?xmltex \floatpos{t}?><table-wrap id="Ch1.T7" specific-use="star"><?xmltex \currentcnt{7}?><label>Table 7</label><caption><p id="d1e3845">Comparing the agreement in the mode for avalanche size estimates with previous studies.</p></caption><oasis:table frame="topbot"><oasis:tgroup cols="5">
     <oasis:colspec colnum="1" colname="col1" align="left"/>
     <oasis:colspec colnum="2" colname="col2" align="right"/>
     <oasis:colspec colnum="3" colname="col3" align="right"/>
     <oasis:colspec colnum="4" colname="col4" align="right"/>
     <oasis:colspec colnum="5" colname="col5" align="right"/>
     <oasis:thead>
       <oasis:row>
         <oasis:entry colname="col1"/>
         <oasis:entry rowsep="1" namest="col2" nameend="col4" align="center">Average agreement (%) </oasis:entry>
         <oasis:entry colname="col5"/>
       </oasis:row>
       <oasis:row>
         <oasis:entry colname="col1">Study</oasis:entry>
         <oasis:entry colname="col2">Full</oasis:entry>
         <oasis:entry colname="col3">Full size</oasis:entry>
         <oasis:entry colname="col4">Intermediate</oasis:entry>
         <oasis:entry colname="col5">Raters</oasis:entry>
       </oasis:row>
       <oasis:row rowsep="1">
         <oasis:entry colname="col1"/>
         <oasis:entry colname="col2">size</oasis:entry>
         <oasis:entry colname="col3"><inline-formula><mml:math id="M180" display="inline"><mml:mo>±</mml:mo></mml:math></inline-formula>1 class</oasis:entry>
         <oasis:entry colname="col4">size</oasis:entry>
         <oasis:entry colname="col5">(samples)</oasis:entry>
       </oasis:row>
     </oasis:thead>
     <oasis:tbody>
       <oasis:row>
         <oasis:entry colname="col1">
                    <xref ref-type="bibr" rid="bib1.bibx33" id="text.60"/>
                  <inline-formula><mml:math id="M181" display="inline"><mml:msup><mml:mi/><mml:mo>*</mml:mo></mml:msup></mml:math></inline-formula></oasis:entry>
         <oasis:entry colname="col2">62</oasis:entry>
         <oasis:entry colname="col3">–</oasis:entry>
         <oasis:entry colname="col4">25</oasis:entry>
         <oasis:entry colname="col5">61 (18)</oasis:entry>
       </oasis:row>
       <oasis:row>
         <oasis:entry colname="col1">
                    <xref ref-type="bibr" rid="bib1.bibx24" id="text.61"/>
                  </oasis:entry>
         <oasis:entry colname="col2">79</oasis:entry>
         <oasis:entry colname="col3">100</oasis:entry>
         <oasis:entry colname="col4">44</oasis:entry>
         <oasis:entry colname="col5">22 (18)</oasis:entry>
       </oasis:row>
       <oasis:row>
         <oasis:entry colname="col1">
                    <xref ref-type="bibr" rid="bib1.bibx20" id="text.62"/>
                  </oasis:entry>
         <oasis:entry colname="col2">84</oasis:entry>
         <oasis:entry colname="col3">–</oasis:entry>
         <oasis:entry colname="col4">–</oasis:entry>
         <oasis:entry colname="col5">2 (351)</oasis:entry>
       </oasis:row>
       <oasis:row>
         <oasis:entry colname="col1">This study</oasis:entry>
         <oasis:entry colname="col2">67</oasis:entry>
         <oasis:entry colname="col3">97</oasis:entry>
         <oasis:entry colname="col4">49</oasis:entry>
         <oasis:entry colname="col5">170 (10)</oasis:entry>
       </oasis:row>
     </oasis:tbody>
   </oasis:tgroup></oasis:table><table-wrap-foot><p id="d1e3848"><inline-formula><mml:math id="M179" display="inline"><mml:msup><mml:mi/><mml:mo>*</mml:mo></mml:msup></mml:math></inline-formula> European forecasters only; for the Canadian ones, see <xref ref-type="bibr" rid="bib1.bibx24" id="text.59"/>.</p></table-wrap-foot><?xmltex \gdef\@currentlabel{7}?></table-wrap>

      <p id="d1e4024">In our survey, we found the lowest agreement with <inline-formula><mml:math id="M182" display="inline"><mml:mrow><mml:msub><mml:mi>s</mml:mi><mml:mi mathvariant="normal">mode</mml:mi></mml:msub></mml:mrow></mml:math></inline-formula> for the three largest avalanches (<inline-formula><mml:math id="M183" display="inline"><mml:mrow><mml:msub><mml:mover accent="true"><mml:mi>s</mml:mi><mml:mo mathvariant="normal">‾</mml:mo></mml:mover><mml:mi mathvariant="normal">j</mml:mi></mml:msub><mml:mo>≥</mml:mo><mml:mn mathvariant="normal">3.5</mml:mn></mml:mrow></mml:math></inline-formula>) (avalanches  in Fig. <xref ref-type="fig" rid="Ch1.F4"/>b, g and i). For these three avalanches, <inline-formula><mml:math id="M184" display="inline"><mml:mrow><mml:msub><mml:mi>s</mml:mi><mml:mi mathvariant="normal">mode</mml:mi></mml:msub></mml:mrow></mml:math></inline-formula> differed between Europe and North America. <xref ref-type="bibr" rid="bib1.bibx24" id="text.63"/> argued that practitioners have more experience with smaller avalanches (sizes 1, 2 and 3), which are much more frequent than larger avalanches, which may cause size estimates of large avalanches to be more variable and less accurate.</p>
      <p id="d1e4073">We noted systematic differences between size estimates provided by North Americans and by Europeans, with North Americans tending towards smaller sizes (Fig. <xref ref-type="fig" rid="Ch1.F5"/>). This might stem from differences in the European and the North American definitions (see Fig. <xref ref-type="fig" rid="Ch1.F5"/>): for the typical length the European definition provides a range, whereas in the North American definition a typical value is given. The European definition encompasses the North American values, describing typical length for the smaller avalanches, while it coincides with the upper bound for size 4 and provides only a minimum value for size 5. Another difference is that the North American definition includes typical mass, in line with the definition introduced by <xref ref-type="bibr" rid="bib1.bibx30" id="text.64"/>, while in Europe typical volume is defined. Combining deposit volume<?pagebreak page2908?> with density measurements or density estimates of the deposit, mass may be determined (mass <inline-formula><mml:math id="M185" display="inline"><mml:mo>=</mml:mo></mml:math></inline-formula> volume <inline-formula><mml:math id="M186" display="inline"><mml:mo>×</mml:mo></mml:math></inline-formula> density). For instance, calculating the mass of avalanches assuming a mean density of 390 kg m<inline-formula><mml:math id="M187" display="inline"><mml:msup><mml:mi/><mml:mrow><mml:mo>-</mml:mo><mml:mn mathvariant="normal">3</mml:mn></mml:mrow></mml:msup></mml:math></inline-formula>, measured from 95 avalanches at Rogers Pass, British Columbia <xref ref-type="bibr" rid="bib1.bibx31" id="paren.65"/>, avalanches are almost 4 times larger in the European compared to the North American definition; 4 times larger corresponds approximately to a half size (e.g., for size 2 the mass according to the definition is <inline-formula><mml:math id="M188" display="inline"><mml:mrow><mml:msup><mml:mn mathvariant="normal">10</mml:mn><mml:mn mathvariant="normal">2</mml:mn></mml:msup><mml:mo>=</mml:mo><mml:mn mathvariant="normal">100</mml:mn></mml:mrow></mml:math></inline-formula> t; for size 2.5, it is <inline-formula><mml:math id="M189" display="inline"><mml:mrow><mml:msup><mml:mn mathvariant="normal">10</mml:mn><mml:mn mathvariant="normal">2.5</mml:mn></mml:msup><mml:mo>=</mml:mo><mml:mn mathvariant="normal">316</mml:mn></mml:mrow></mml:math></inline-formula> t). Consequently, the significant intercontinental differences may be, at least partially, attributed to the differing size class definitions.</p>
      <p id="d1e4143">We also observed differences in the importance ranking assigned to the factors determining avalanche size, with both the criteria runout and volume being considered more relevant for size estimation by North Americans compared to Europeans. This is particularly noteworthy as neither a description of runout nor an indication of volume are part of the North American size definitions (Table <xref ref-type="table" rid="Ch1.T1"/>). We found the size-determining factor of destructive potential to be considered<?pagebreak page2909?> the least important by North Americans and the second-least important by Europeans. This is surprising as both the definitions in Europe and North America state that avalanche size is classified according to destructive potential <xref ref-type="bibr" rid="bib1.bibx14 bib1.bibx11" id="paren.66"><named-content content-type="pre">e.g.,</named-content></xref>. Furthermore, this finding is also contradictory to that in the study by <xref ref-type="bibr" rid="bib1.bibx33" id="text.67"/>, where destructive potential was the highest-rated factor. One possible reason for the low importance of destructive potential in our study might be related to the study design, where supplementary information about damage to property or people, beyond the photographs, was absent. Thus, destructive potential had to be inferred from avalanche properties like width, length and volume, which is arguably the normal situation when estimating avalanche size.</p>
      <p id="d1e4156">To find out if the way an avalanche is shown in an image influences size estimations, we included one avalanche twice, changing the perspective and zoom. Even though 78 % rated the avalanche in Fig. <xref ref-type="fig" rid="Ch1.F4"/>a and c the same, we observed a significant proportion of larger estimates in the close-up view. We suspect that this might be caused by the perception of the avalanche being larger when covering more area in the photograph. But our sample is small, and understanding the effect of perspective and area covered by avalanche would require further investigations with a more meaningful sample.</p>
</sec>
<sec id="Ch1.S5.SS2">
  <label>5.2</label><title>Avalanche mapping from oblique photographs (study 2) and from remotely sensed imagery (study 3)</title>
      <p id="d1e4169">Study 2 required participants to first find the location of the avalanche on the map, matching the topography visible in the images with the topography as shown on the map, before mapping was possible. The 10 experts, all very familiar with the study area, located the avalanches in the same place. This first step of the assignment would have been more difficult for someone that did not know the area well, possibly resulting in entirely different locations and hence mappings. Thus, the mean overlapping proportion of any two mappings (IoU<inline-formula><mml:math id="M190" display="inline"><mml:msub><mml:mi/><mml:mi mathvariant="normal">pairwise</mml:mi></mml:msub></mml:math></inline-formula>), which was 0.52 in our study, may potentially be 0 if an avalanche is located in the wrong place. We therefore assume that an IoU<inline-formula><mml:math id="M191" display="inline"><mml:msub><mml:mi/><mml:mi mathvariant="normal">pairwise</mml:mi></mml:msub></mml:math></inline-formula> of 0.52 may well describe the upper limit of agreement in mappings from oblique photographs.</p>
      <p id="d1e4190">Study 2 allowed for a comparison with a reference mapping using a methodology superior to the approach the 10 experts used. Thus, the agreement between experts' mapping and the reference mapping can be interpreted as the proportion correct and, hence, allows for assessing the experts' competence in mapping avalanche outlines. The overall proportion considered correct (IoU<inline-formula><mml:math id="M192" display="inline"><mml:msub><mml:mi/><mml:mi mathvariant="normal">ref</mml:mi></mml:msub></mml:math></inline-formula>) was 0.6, with a clear bias towards smaller mapped areas compared to the reference. The results showed that experts were not equally competent, with the proportion correct ranging from 0.4 to 0.8 (Table <xref ref-type="table" rid="Ch1.T4"/>). The agreement between individual mappings and the reference mapping is larger than the agreement in the mappings between participants (IoU<inline-formula><mml:math id="M193" display="inline"><mml:mrow><mml:msub><mml:mi/><mml:mi mathvariant="normal">pairwise</mml:mi></mml:msub><mml:mo>=</mml:mo><mml:mn mathvariant="normal">0.52</mml:mn></mml:mrow></mml:math></inline-formula>). This means that the reliability of individual mappings would be underestimated when relying on a measure like the agreement rate between domain experts. If competence is known, it would be possible to weigh individual mappings if two or more mappings were available, likely resulting in more reliable results. Overall, we consider the mapping of avalanches using oblique photographs a challenging task to perform consistently and accurately.</p>
      <p id="d1e4218">In study 3, five participants had to identify the avalanches in the remotely sensed imagery, for each of four image resolutions, before mapping them. In other words, whether a point is identified as an avalanche is a combination of existential and extensional uncertainty <xref ref-type="bibr" rid="bib1.bibx32" id="paren.68"/>, addressing the following questions: is there an avalanche, and where are the boundaries? This uncertainty was lower with higher image resolution, and for illuminated compared to shaded parts of the image, it allowed participants to identify avalanche area more consistently and confirmed the findings of earlier work <xref ref-type="bibr" rid="bib1.bibx21" id="paren.69"/>. Moreover, snow conditions influenced agreement too, with higher agreement under wet-snow compared to dry-snow conditions (Fig. <xref ref-type="fig" rid="Ch1.F10"/>). We suspect that this difference is caused by the presence of liquid water in the case of a wet snowpack, which leads to more pronounced avalanche boundaries compared to dry-snow conditions.</p>
      <p id="d1e4229">Another important finding from these studies are the large differences in the areas mapped as an avalanche by the experts. For the six avalanches in study 2, the largest mapped area was between 2 and 4 times larger than the smallest mapped area (Table <xref ref-type="fig" rid="Ch1.F7"/>). For study 3, variations across resolutions were found to be smaller than the variations in IoU<inline-formula><mml:math id="M194" display="inline"><mml:msub><mml:mi/><mml:mi mathvariant="normal">pairwise</mml:mi></mml:msub></mml:math></inline-formula> per resolution, suggesting individual experience and competence have a larger impact than the underlying spatial resolution. If avalanches would be classified automatically using area or by extracting width and length from the mapping <xref ref-type="bibr" rid="bib1.bibx36" id="paren.70"><named-content content-type="pre">e.g.,</named-content></xref>, completely different size classes may result due to these variations. For instance, comparing the mappings of 4000 avalanches with the reported size estimate, <xref ref-type="bibr" rid="bib1.bibx46" id="text.71"/> showed that the median area of size-2 avalanches was about 3 to 5 times larger than size-1 avalanches or that the mapped area of size-4 avalanches was about 7 times larger compared to size-3 avalanches <xref ref-type="bibr" rid="bib1.bibx46" id="paren.72"><named-content content-type="post">pp. 49 and 51</named-content></xref>. Comparing these values to the variation observed in the mappings by different experts in our study suggests that estimating avalanche size based on mapped area would, quite frequently, result in different size estimates.</p>
</sec>
<sec id="Ch1.S5.SS3">
  <label>5.3</label><title>Implications for practice</title>
      <p id="d1e4264">Several sources of error may impact the reliability of tasks involving human judgment and estimation. These are related to the data being suitable; the skill or competence of the rater interpreting the data; and, finally, the level of generalization<?pagebreak page2910?> (e.g., forcing avalanches into five size classes). In the context of size estimation and outline determination, low image resolution, lack of image context/reference objects and unfavorable illumination conditions such as shade or diffuse light may impact the conclusiveness of the data for the task to be performed and, hence, reliability. Similarly, differences in the raters' backgrounds and experiences may lead to different levels of competence in interpreting the data. Furthermore, variations in the perception and definition may result in systematic variations in size estimates or outlines. Moreover, the binary choice between an avalanche and no avalanche, when mapping avalanche outlines, does not allow for expressing uncertainty. This may lead to more pronounced differences in the avalanche area identified by participants. Addressing and being aware of these issues will improve the quality of avalanche size estimation and outline determination.</p>
      <p id="d1e4267">The results of this study indicate that size inventories from North America and Europe, different warning services within Europe, or simply different domain experts may systematically differ in their assessments. Consequently, transferability of size inventories between continents or different warning services may be limited. To achieve a common understanding and comparable size estimation, in particular for expert forecasters and for observers, we suggest a joint effort of the continental and national avalanche associations, together with avalanche forecasters and other avalanche practitioners, to develop training tools that help standardize the size estimations. An in-depth analysis of current protocols or training programs could be fruitful and serve as a first step in tackling this issue. One option might be training people in the “all observables approach”  advocated by <xref ref-type="bibr" rid="bib1.bibx30" id="text.73"/>, which requires imagining the objects that might be destroyed in the track or start of the runout zone of an avalanche.</p>
      <p id="d1e4273">In the meantime, the uncertainty related to size estimates may be reduced taking into account second estimates and/or jointly deciding on the size in the case of disagreements <xref ref-type="bibr" rid="bib1.bibx20" id="paren.74"><named-content content-type="pre">e.g.,</named-content></xref>. Additionally, we recommend the use of intermediate sizes in the following way: first the (full) size class should be estimated. In a second step, the assessor may judge whether avalanche size is low or high or in the middle of the class <xref ref-type="bibr" rid="bib1.bibx18" id="paren.75"><named-content content-type="pre">like suggested for evaluative social judgments by</named-content></xref>. If it was low or high, this would result in the intermediate sizes between the chosen and the upper or lower adjacent full size. Practically, this could mean that from a full size 2 the assessor may, in a second step, assign size 1.5 if the avalanche is at the lower end of size 2, assign 2.5 if it is at the upper end or keep a full size 2 if it is a “typical” avalanche for that size.</p>
      <p id="d1e4286">While the observed variations in avalanche outlines may partly be attributed to a different background and level of experience, we argue that it is partly caused by the lack of a common, precise definition of where to delimit an avalanche. We are not aware of any unambiguous, actionable guideline of where exactly to place the visible outline. Arguably, there is no “natural”, self-evident definition, especially for dry-snow avalanches. Consistency, in the sense of repeatability across expert annotators, can perhaps only be achieved through a generally agreed consensus that includes shared but, to some degree, arbitrary conventions. It appears that a standardization effort may be beneficial and that standardized training could go a long way towards reducing the spread between different experts and organizations, even if some causes of variability, e.g., lighting conditions after a large snowfall cannot easily be controlled and will remain.</p>
      <p id="d1e4290">If reliable mappings from photographs are required, we recommend second mappings, jointly deciding on the extent of the outline, using a monoplotting tool <xref ref-type="bibr" rid="bib1.bibx7" id="paren.76"><named-content content-type="pre">e.g.,</named-content></xref> or the overlay image capabilities of Google Earth. For remotely sensed imagery we advocate using a spatial resolution of 50 cm or finer for the detailed segmentation of specific avalanches, whereas a resolution of approximately 2 m seems to be sufficient to capture the overall avalanche activity over a larger region. Intermediate resolutions may provide a reasonable compromise between the precision of individual outlines and large-area coverage at a reasonable cost and immediacy. Recording the perceived uncertainty while mapping might help <xref ref-type="bibr" rid="bib1.bibx21" id="paren.77"/> as well as using the area of agreement from several mappings or jointly discussing areas of disagreement. We generally recommend aiming for good illumination for mapping avalanches, especially under dry-snow conditions.</p>
</sec>
<sec id="Ch1.S5.SS4">
  <label>5.4</label><title>Limitations</title>
      <p id="d1e4310">In all three studies, we relied on comparably small sets of images: in study 1, the size survey, we aimed at a high response rate, which came at the cost of a smaller selection of different avalanche examples (10) since we did not want to introduce a bias by showing different images to different participants. The two mapping tasks (studies 2 and 3) were rather time-consuming and were, therefore, limited to few participants and to few examples. They may therefore be regarded as pilot studies, whose results should be interpreted with caution, keeping in mind the comparably small datasets and the potential particularities of the data. In particular, biases may be present due to the homogeneous sample from participants with a similar background and training, as well as due to the chosen oblique photos and remotely sensed data.</p>
      <p id="d1e4313">For study 1, we provided photographs but no maps and no additional information, for instance on damage which may have occurred. This certainly made the size estimation task somewhat more difficult, as we often suspect either that a map is available or that the person is familiar with the avalanche path, which may both help in estimating avalanche dimensions. We did not provide maps together with the photographs, as we wanted to avoid introducing a bias related to the (un)familiarity with a specific map design. We have, however, tried to compensate the lack of an accompanying map through the presence of reference objects (people, trees,<?pagebreak page2911?> ski lifts, etc.) in our example photographs to help participants determine the potential damage and volume/mass. Nevertheless, photographs in Fig. <xref ref-type="fig" rid="Ch1.F4"/>a and e lack recognizable reference objects, and the other photographs (Fig. <xref ref-type="fig" rid="Ch1.F4"/>) must be analyzed carefully to identify the ski tracks, people and houses. Agreement for avalanches next to clearly visible reference objects could therefore be higher than in our analysis.</p>
      <p id="d1e4320">For the avalanches mapped from oblique photographs (study 2), we speculate that study participants, being aware that their mappings will be analyzed in detail, may have paid more attention to finding the exact boundaries than during routine documentation work. We acknowledge that our sample size of six avalanche examples covers only a fraction of possible viewing angles, snow and avalanche conditions, and terrain characteristics. Furthermore, all participants were well acquainted with the study area and had extensive experience with mapping avalanches using oblique photos.
Thus, we regard our results as a best-case scenario. Still, we believe that within the range of (fairly typical) conditions captured by our set of pictures, the evaluation is representative of avalanche outlines currently used in Switzerland. We encourage further research to ascertain the worldwide validity of the results.</p>
      <p id="d1e4323">Finally, we would like to point out that three of the authors were also involved as participants in the studies (one in study 1 and two in studies 2 and 3). Particularly in studies 2 and 3, with few participants, this may impact results favorably and suggests the presented findings should be treated as an upper bound.</p>
</sec>
</sec>
<sec id="Ch1.S6" sec-type="conclusions">
  <label>6</label><title>Conclusions and outlook</title>
      <p id="d1e4335">We quantified uncertainty related to avalanche size estimation and avalanche outline determination, calculating the proportion of agreement between raters, the agreement with the most frequently chosen size and the agreement with the reference size. For avalanche outlines we investigated spatial agreement using the intersection over union between individual mappings as well as compared to a reference. Like in <xref ref-type="bibr" rid="bib1.bibx44" id="text.78"/> the amount of variation depends on the type of task presented to the operator: we showed that it is difficult to consistently estimate avalanche sizes, and our analyses revealed significant differences between North American and European experts. The mapping of avalanches either from oblique photographs or from remotely sensed imagery proved to be a challenging task resulting in large intra-rater variabilities: some experts showed consistently larger deviations from the reference data. In most extreme case this resulted in the deviation being 2 to 4 times larger than the smallest mapped avalanche area. For the mapping from remotely sensed imagery, individual experience and competence proved to have a larger impact than the underlying spatial resolution. Snow conditions also influenced agreement, with higher agreement under wet-snow compared to dry-snow conditions. For both mapping tasks our samples were fairly homogeneous and the sample size was rather small, limiting the generalizability of our results. Nevertheless, they shed light on the uncertainty underlying avalanche outlines for the first time. We strongly encourage further investigations into agreement between avalanche outlines to paint a more complete picture of the variation in the currently widely used and generated datasets.</p>
      <p id="d1e4341">Our findings indicate that the reliability of human estimates as a reference or ground truth for avalanche-related tasks needs to be questioned and critically assessed. Since these data are used as ground truth, for instance, for the validation of numerical avalanche simulations <xref ref-type="bibr" rid="bib1.bibx48" id="paren.79"><named-content content-type="pre">e.g.,</named-content></xref> or for training models estimating avalanche size from snowpack simulations <xref ref-type="bibr" rid="bib1.bibx29" id="paren.80"><named-content content-type="pre">e.g.,</named-content></xref>, efforts should be made to obtain at least an approximate idea on the reliability of labels used when depending on them. Specifically efforts to average out unsystematic error and those requiring justification for the choice to endorse the analytic process <xref ref-type="bibr" rid="bib1.bibx38 bib1.bibx22" id="paren.81"/> may help to achieve more reliable results for the avalanche-related tasks presented in this paper. This could be achieved by relying on a superior method to obtain a ground truth or otherwise independent estimates of several experts to allow, for example, choosing the most frequent size.</p>
      <p id="d1e4357">Besides suggesting more precise definitions and training protocols, our results call for automation. Modern image analysis algorithms, often based on machine learning <xref ref-type="bibr" rid="bib1.bibx21" id="paren.82"><named-content content-type="pre">like</named-content><named-content content-type="post">in the context of avalanche mapping from SPOT 6 or 7 – Satellite pour l'Observation de la Terre – imagery</named-content></xref>, are by no means perfect, but they rival human performance and offer consistent, repeatable results. Our reliability study may serve as a baseline to relate the outputs of such automatic methods to human expert performance. Even though the models cannot erase the inter-observer variability and will only learn to reproduce the outlines they are trained with, they can help to generate reproducible and comparable results.</p><?xmltex \hack{\clearpage}?>
</sec>

      
      </body>
    <back><app-group>

<?pagebreak page2912?><app id="App1.Ch1.S1">
  <?xmltex \currentcnt{A}?><label>Appendix A</label><title/>

      <?xmltex \floatpos{h!}?><fig id="App1.Ch1.S1.F11"><?xmltex \currentcnt{A1}?><?xmltex \def\figurename{Figure}?><label>Figure A1</label><caption><p id="d1e4380">IoU for all expert pairs for the mapping from oblique photographs. The numbers I to X represent the different expert participants.</p></caption>
        <?xmltex \hack{\hsize\textwidth}?>
        <?xmltex \igopts{width=369.885827pt}?><graphic xlink:href="https://nhess.copernicus.org/articles/23/2895/2023/nhess-23-2895-2023-f09.png"/>

      </fig>

</app>
  </app-group><notes notes-type="dataavailability"><title>Data availability</title>

      <p id="d1e4395">The survey results, used oblique photographs and remotely sensed imagery as well as all mapped avalanche outlines are available on EnviDat (<ext-link xlink:href="https://doi.org/10.16904/envidat.423" ext-link-type="DOI">10.16904/envidat.423</ext-link>, <xref ref-type="bibr" rid="bib1.bibx19" id="altparen.83"/>).</p>
  </notes><app-group>
        <supplementary-material position="anchor"><p id="d1e4404">The supplement related to this article is available online at: <inline-supplementary-material xlink:href="https://doi.org/10.5194/nhess-23-2895-2023-supplement" xlink:title="pdf">https://doi.org/10.5194/nhess-23-2895-2023-supplement</inline-supplementary-material>.</p></supplementary-material>
        </app-group><notes notes-type="authorcontribution"><title>Author contributions</title>

      <p id="d1e4413">EDH designed the size survey with input from FT, initiated and coordinated the avalanche mappings, performed the analyses in close collaboration with the co-authors, and wrote the paper draft. FT and EDH were part of the team of experts mapping from photographs. EDH and YB were part of the team of experts mapping from remotely sensed imagery. FT delivered the necessary input from the avalanche warning service, critically reviewed the results, and heavily contributed to the analysis and writing of the paper. All co-authors reviewed and complemented the manuscript.</p>
  </notes><notes notes-type="competinginterests"><title>Competing interests</title>

      <p id="d1e4419">At least one of the (co-)authors is a member of the editorial board of <italic>Natural Hazards and Earth System Sciences</italic>.</p>
  </notes><?xmltex \hack{\newpage}?><?xmltex \hack{~\\[101mm]}?><notes notes-type="disclaimer"><title>Disclaimer</title>

      <p id="d1e4431">Publisher’s note: Copernicus Publications remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.</p>
  </notes><ack><title>Acknowledgements</title><p id="d1e4437">We thank Alessio Krenger, Jessica Munch and Tatjana Scherrer for helping with the translation of the surveys into Italian and French. We are grateful for the large number of avalanche forecasters from North America and Europe who took the time to fill out our survey. We thank Silke Griesser, Mark Schaer, Laura Stephan, Lukas Stoffel, Thomas Stucki, Jürg Trachsel and Kurt Winkler for taking part in the avalanche mapping from photographs. We are thankful for the contribution of Leon Bührle, Lucien Oberson and Christina Salzmann in the mapping of all visible avalanches for all four resolutions in the airplane and drone data. We are grateful to Gwendolyn Dasser for the valuable and constructive feedback, which helped to improve the paper. We thank Ivan Moner and Brian Lazar for the critical questions, suggestions and comments in their reviews.</p></ack><notes notes-type="financialsupport"><title>Financial support</title>

      <p id="d1e4442">The acquisition of the data used in study 3 has been partially supported by the Swiss National Science Foundation (SNF; grant no. 200021_172800).</p>
  </notes><notes notes-type="reviewstatement"><title>Review statement</title>

      <p id="d1e4448">This paper was edited by Pascal Haegeli and reviewed by Ivan Moner and Brian Lazar.</p>
  </notes><ref-list>
    <title>References</title>

      <ref id="bib1.bibx1"><?xmltex \def\ref@label{{{American Avalanche Association}(2022)}}?><label>American Avalanche Association(2022)</label><?label AmericanAvalAssoc.2022?><mixed-citation>
American Avalanche Association: Snow, Weather and Avalanches: Observation
Guidelines for Avalanche Programs in the United States, 4th edn.,
ISBN 979-8-218-05765-7, 2022.</mixed-citation></ref>
      <ref id="bib1.bibx2"><?xmltex \def\ref@label{{Ardizzone et~al.(2002)Ardizzone, Cardinali, Carrara, Guzzetti, and
Reichenbach}}?><label>Ardizzone et al.(2002)Ardizzone, Cardinali, Carrara, Guzzetti, and
Reichenbach</label><?label ardizzone.2002?><mixed-citation>Ardizzone, F., Cardinali, M., Carrara, A., Guzzetti, F., and Reichenbach, P.: Impact of mapping errors on the reliability of landslide hazard maps, Nat. Hazards Earth Syst. Sci., 2, 3–14, <ext-link xlink:href="https://doi.org/10.5194/nhess-2-3-2002" ext-link-type="DOI">10.5194/nhess-2-3-2002</ext-link>, 2002.</mixed-citation></ref>
      <ref id="bib1.bibx3"><?xmltex \def\ref@label{{Bianchi et~al.(2021)Bianchi, Grahn, Eckerstorfer, Malnes, and
Vickers}}?><label>Bianchi et al.(2021)Bianchi, Grahn, Eckerstorfer, Malnes, and
Vickers</label><?label bianchi.2021?><mixed-citation>Bianchi, F. M., Grahn, J., Eckerstorfer, M., Malnes, E., and Vickers, H.: Snow
Avalanche Segmentation in SAR Images With Fully Convolutional Neural
Networks, IEEE J. Sel. Top. Appl., 14, 75–82, <ext-link xlink:href="https://doi.org/10.1109/JSTARS.2020.3036914" ext-link-type="DOI">10.1109/JSTARS.2020.3036914</ext-link>, 2021.</mixed-citation></ref>
      <ref id="bib1.bibx4"><?xmltex \def\ref@label{{Birkeland and Greene(2011)}}?><label>Birkeland and Greene(2011)</label><?label birkelandgreene.2011?><mixed-citation>Birkeland, K. and Greene, E.: Accurately Assessing Avalanche Size: The Ins and
Outs of the R- and D- scales, The Avalanche Review, 29, 27/32,
<uri>https://avalanche.org/wp-content/uploads/2018/08/11_TAR_BirkelandGreene.pdf</uri> (last access: 5 January 2023),
2011.</mixed-citation></ref>
      <ref id="bib1.bibx5"><?xmltex \def\ref@label{{Bowler(2006)}}?><label>Bowler(2006)</label><?label bowler.2006?><mixed-citation>Bowler, N. E.: Explicitly Accounting for Observation Error in Categorical
Verification of Forecasts, Mon. Weather Rev., 134, 1600–1606,
<ext-link xlink:href="https://doi.org/10.1175/MWR3138.1" ext-link-type="DOI">10.1175/MWR3138.1</ext-link>, 2006.</mixed-citation></ref>
      <ref id="bib1.bibx6"><?xmltex \def\ref@label{{Bozzini et~al.(2012)Bozzini, Conedera, and Krebs}}?><label>Bozzini et al.(2012)Bozzini, Conedera, and Krebs</label><?label bozzini.2012?><mixed-citation>Bozzini, C., Conedera, M., and Krebs, P.: A New Monoplotting Tool to Extract
Georeferenced Vector Data and Orthorectified Raster Data from Oblique
Non-Metric Photographs, International Journal of Heritage in the Digital Era,
1, 499–518, <ext-link xlink:href="https://doi.org/10.1260/2047-4970.1.3.499" ext-link-type="DOI">10.1260/2047-4970.1.3.499</ext-link>, 2012.</mixed-citation></ref>
      <ref id="bib1.bibx7"><?xmltex \def\ref@label{{Bozzini et~al.(2013)Bozzini, Conedera, and Krebs}}?><label>Bozzini et al.(2013)Bozzini, Conedera, and Krebs</label><?label bozzini.2013?><mixed-citation>Bozzini, C., Conedera, M., and Krebs, P.: A new tool for facilitating the
retrieval and recording of the place name cultural heritage, The
International Archives of the Photogrammetry, Remote Sensing and Spatial
Information Sciences, XL-5/W2, 115–118,
<ext-link xlink:href="https://doi.org/10.5194/isprsarchives-XL-5-W2-115-2013" ext-link-type="DOI">10.5194/isprsarchives-XL-5-W2-115-2013</ext-link>, 2013.</mixed-citation></ref>
      <ref id="bib1.bibx8"><?xmltex \def\ref@label{{Brardinoni et~al.(2019)Brardinoni, Scotti, Sailer, and
Mair}}?><label>Brardinoni et al.(2019)Brardinoni, Scotti, Sailer, and
Mair</label><?label brardinoni.2019?><mixed-citation>Brardinoni, F., Scotti, R., Sailer, R., and Mair, V.: Evaluating sources of
uncertainty and variability in rock glacier inventories, Earth Surf.
Proc. Land., 44, 2450–2466,
<ext-link xlink:href="https://doi.org/10.1002/esp.4674" ext-link-type="DOI">10.1002/esp.4674</ext-link>, 2019.</mixed-citation></ref>
      <ref id="bib1.bibx9"><?xmltex \def\ref@label{{B{\"{u}}hler et~al.(2019)B{\"{u}}hler, Hafner, Zweifel, Zesiger, and
Heisig}}?><label>Bühler et al.(2019)Bühler, Hafner, Zweifel, Zesiger, and
Heisig</label><?label buhler.2019?><mixed-citation>Bühler, Y., Hafner, E. D., Zweifel, B., Zesiger, M., and Heisig, H.: Where are the avalanches? Rapid SPOT6 satellite data acquisition to map an extreme avalanche period over the Swiss Alps, The Cryosphere, 13, 3225–3238, <ext-link xlink:href="https://doi.org/10.5194/tc-13-3225-2019" ext-link-type="DOI">10.5194/tc-13-3225-2019</ext-link>, 2019.</mixed-citation></ref>
      <ref id="bib1.bibx10"><?xmltex \def\ref@label{{B\"{u}hler et~al.(2022)B\"{u}hler, Bebi, Christen, Margreth, Stoffel,
Stoffel, Marty, Schmucki, Caviezel, K\"{u}hne, Wohlwend, and
Bartelt}}?><label>Bühler et al.(2022)Bühler, Bebi, Christen, Margreth, Stoffel,
Stoffel, Marty, Schmucki, Caviezel, Kühne, Wohlwend, and
Bartelt</label><?label buhler.2022?><mixed-citation>Bühler, Y., Bebi, P., Christen, M., Margreth, S., Stoffel, L., Stoffel, A., Marty, C., Schmucki, G., Caviezel, A., Kühne, R., Wohlwend, S., and Bartelt, P.: Automated avalanche hazard indication mapping on a statewide scale, Nat. Hazards Earth Syst. Sci., 22, 1825–1843, <ext-link xlink:href="https://doi.org/10.5194/nhess-22-1825-2022" ext-link-type="DOI">10.5194/nhess-22-1825-2022</ext-link>, 2022.</mixed-citation></ref>
      <ref id="bib1.bibx11"><?xmltex \def\ref@label{{{CAA}(2023)}}?><label>CAA(2023)</label><?label can.2023?><mixed-citation>CAA: Avalanche Size,
<uri>https://www.avalanche-center.org/Education/glossary/avalanche-size.php</uri> (last access: 4 January 2023),
2023.</mixed-citation></ref>
      <ref id="bib1.bibx12"><?xmltex \def\ref@label{{{Canadian Avalanche Association}(2016)}}?><label>Canadian Avalanche Association(2016)</label><?label CanadianAvalAssoc.2016?><mixed-citation>Canadian Avalanche Association: Observation guidelines and recording
standards for weather, snowpack and avalanches,
<uri>https://cdn.ymaws.com/www.avalancheassociation.ca/resource/resmgr/standards_docs/OGRS2016web.pdf</uri> (last access: 25 August 2023),
2016.</mixed-citation></ref>
      <ref id="bib1.bibx13"><?xmltex \def\ref@label{{Cronbach(1947)}}?><label>Cronbach(1947)</label><?label Cronbach.1947?><mixed-citation>Cronbach, L. J.: Test “reliability”: Its
meaning and determination, Psychometrika, 12, 1–16,
<ext-link xlink:href="https://doi.org/10.1007/bf02289289" ext-link-type="DOI">10.1007/bf02289289</ext-link>, 1947.</mixed-citation></ref>
      <ref id="bib1.bibx14"><?xmltex \def\ref@label{{EAWS(2023)}}?><label>EAWS(2023)</label><?label eaws.2023?><mixed-citation>EAWS: Standards: Avalanche Size,
<uri>https://www.avalanches.org/standards/avalanche-size/</uri> (last
access: 8 May 2023), 2023.</mixed-citation></ref><?xmltex \hack{\newpage}?>
      <ref id="bib1.bibx15"><?xmltex \def\ref@label{{Eckerstorfer et~al.(2016)Eckerstorfer, Bühler, Frauenfelder, and
Malnes}}?><label>Eckerstorfer et al.(2016)Eckerstorfer, Bühler, Frauenfelder, and
Malnes</label><?label eckerstorfer.2016?><mixed-citation>Eckerstorfer, M., Bühler, Y., Frauenfelder, R., and Malnes, E.: Remote sensing
of snow avalanches: Recent advances, potential, and limitations, Cold Reg.
Sci. Technol., 121, 126–140,
<ext-link xlink:href="https://doi.org/10.1016/j.coldregions.2015.11.001" ext-link-type="DOI">10.1016/j.coldregions.2015.11.001</ext-link>, 2016.</mixed-citation></ref>
      <ref id="bib1.bibx16"><?xmltex \def\ref@label{{Eckerstorfer et~al.(2017)Eckerstorfer, Malnes, and
M{\"{u}}ller}}?><label>Eckerstorfer et al.(2017)Eckerstorfer, Malnes, and
Müller</label><?label eckerstorfer.2017?><mixed-citation>Eckerstorfer, M., Malnes, E., and Müller, K.: A complete snow avalanche
activity record from a Norwegian forecasting region using Sentinel-1
satellite-radar data, Cold Reg. Sci. Technol., 144, 39–51,
<ext-link xlink:href="https://doi.org/10.1016/j.coldregions.2017.08.004" ext-link-type="DOI">10.1016/j.coldregions.2017.08.004</ext-link>, 2017.</mixed-citation></ref>
      <ref id="bib1.bibx17"><?xmltex \def\ref@label{{Galli et~al.(2008)Galli, Ardizzone, Cardinali, Guzzetti, and
Reichenbach}}?><label>Galli et al.(2008)Galli, Ardizzone, Cardinali, Guzzetti, and
Reichenbach</label><?label galli.2008?><mixed-citation>Galli, M., Ardizzone, F., Cardinali, M., Guzzetti, F., and Reichenbach, P.:
Comparing landslide inventory maps, Geomorphology, 94, 268–289,
<ext-link xlink:href="https://doi.org/10.1016/j.geomorph.2006.09.023" ext-link-type="DOI">10.1016/j.geomorph.2006.09.023</ext-link>, 2008.</mixed-citation></ref>
      <ref id="bib1.bibx18"><?xmltex \def\ref@label{{Goffin and Olson(2011)}}?><label>Goffin and Olson(2011)</label><?label goffin.2011?><mixed-citation>Goffin, R. D. and Olson, J. M.: Is It All Relative?: Comparative Judgments and
the Possible Improvement of Self-Ratings and Ratings of Others, Perspect. Psychol. Sci., 6, 48–60, <ext-link xlink:href="https://doi.org/10.1177/1745691610393521" ext-link-type="DOI">10.1177/1745691610393521</ext-link>, 2011.</mixed-citation></ref>
      <ref id="bib1.bibx19"><?xmltex \def\ref@label{{Hafner(2023)}}?><label>Hafner(2023)</label><?label hafner.2023?><mixed-citation>Hafner, E. D.: Data reliability study: avalanche size estimates and outlines, Envidat [data set],
<ext-link xlink:href="https://doi.org/10.16904/envidat.423" ext-link-type="DOI">10.16904/envidat.423</ext-link>, 2023.</mixed-citation></ref>
      <ref id="bib1.bibx20"><?xmltex \def\ref@label{{Hafner et~al.(2021)Hafner, Techel, Leinss, and
B\"{u}hler}}?><label>Hafner et al.(2021)Hafner, Techel, Leinss, and
Bühler</label><?label hafner.2021?><mixed-citation>Hafner, E. D., Techel, F., Leinss, S., and Bühler, Y.: Mapping avalanches with satellites – evaluation of performance and completeness, The Cryosphere, 15, 983–1004, <ext-link xlink:href="https://doi.org/10.5194/tc-15-983-2021" ext-link-type="DOI">10.5194/tc-15-983-2021</ext-link>, 2021.</mixed-citation></ref>
      <ref id="bib1.bibx21"><?xmltex \def\ref@label{{Hafner et~al.(2022)Hafner, Barton, Daudt, Wegner, Schindler, and
B\"{u}hler}}?><label>Hafner et al.(2022)Hafner, Barton, Daudt, Wegner, Schindler, and
Bühler</label><?label hafner.2022?><mixed-citation>Hafner, E. D., Barton, P., Daudt, R. C., Wegner, J. D., Schindler, K., and Bühler, Y.: Automated avalanche mapping from SPOT 6/7 satellite imagery with deep learning: results, evaluation, potential and limitations, The Cryosphere, 16, 3517–3530, <ext-link xlink:href="https://doi.org/10.5194/tc-16-3517-2022" ext-link-type="DOI">10.5194/tc-16-3517-2022</ext-link>, 2022.</mixed-citation></ref>
      <ref id="bib1.bibx22"><?xmltex \def\ref@label{{Hagafors and Brehmer(1983)}}?><label>Hagafors and Brehmer(1983)</label><?label hagfors.1983?><mixed-citation>Hagafors, R. and Brehmer, B.: Does having to justify one's judgments change the
nature of the judgment process?, Organ. Behav. Hum.
Perf., 31, 223–232,
<ext-link xlink:href="https://doi.org/10.1016/0030-5073(83)90122-8" ext-link-type="DOI">10.1016/0030-5073(83)90122-8</ext-link>, 1983.</mixed-citation></ref>
      <ref id="bib1.bibx23"><?xmltex \def\ref@label{{Hendrikx et~al.(2005)Hendrikx, Owens, Carran, and
Carran}}?><label>Hendrikx et al.(2005)Hendrikx, Owens, Carran, and
Carran</label><?label hendrikx.2005?><mixed-citation>
Hendrikx, J., Owens, I., Carran, W., and Carran, A.: Avalanche activity in an
extreme maritime climate: The application of classification trees for
forecasting, Cold Reg. Sci. Technol., 43, 104–116, 2005.</mixed-citation></ref>
      <ref id="bib1.bibx24"><?xmltex \def\ref@label{{Jamieson et~al.(2014)Jamieson, Beglinger, and Wilson}}?><label>Jamieson et al.(2014)Jamieson, Beglinger, and Wilson</label><?label jamieson.2014?><mixed-citation>
Jamieson, B., Beglinger, R., and Wilson, D.: Case study of a large snow
avalanche in the Selkirk Mountains and reflections on the Canadian size
classification, Geohazards 6, 6th Canadian GeoHazards Conference, Kingston,
ON, Canada, 15–18 June 2014, 2014.</mixed-citation></ref>
      <ref id="bib1.bibx25"><?xmltex \def\ref@label{{Korzeniowska et~al.(2017)Korzeniowska, B{\"{u}}hler, Marty, and
Korup}}?><label>Korzeniowska et al.(2017)Korzeniowska, Bühler, Marty, and
Korup</label><?label korzeniowska.2017?><mixed-citation>Korzeniowska, K., Bühler, Y., Marty, M., and Korup, O.: Regional snow-avalanche detection using object-based image analysis of near-infrared aerial imagery, Nat. Hazards Earth Syst. Sci., 17, 1823–1836, <ext-link xlink:href="https://doi.org/10.5194/nhess-17-1823-2017" ext-link-type="DOI">10.5194/nhess-17-1823-2017</ext-link>, 2017.</mixed-citation></ref>
      <ref id="bib1.bibx26"><?xmltex \def\ref@label{{Lampert et~al.(2016)Lampert, Stumpf, and Gancarski}}?><label>Lampert et al.(2016)Lampert, Stumpf, and Gancarski</label><?label lampert.2016?><mixed-citation>
Lampert, T., Stumpf, A., and Gancarski, P.: An empirical study into annotator
agreement, ground truth estimation, and algorithm evaluation, IEEE
T. Image Process., 25, 2557–2572, 2016.</mixed-citation></ref>
      <ref id="bib1.bibx27"><?xmltex \def\ref@label{{Lato et~al.(2012)Lato, Frauenfelder, and B{\"{u}}hler}}?><label>Lato et al.(2012)Lato, Frauenfelder, and Bühler</label><?label lato.2012?><mixed-citation>Lato, M. J., Frauenfelder, R., and Bühler, Y.: Automated detection of snow avalanche deposits: segmentation and classification of optical remote sensing imagery, Nat. Hazards Earth Syst. Sci., 12, 2893–2906, <ext-link xlink:href="https://doi.org/10.5194/nhess-12-2893-2012" ext-link-type="DOI">10.5194/nhess-12-2893-2012</ext-link>, 2012.</mixed-citation></ref>
      <ref id="bib1.bibx28"><?xmltex \def\ref@label{{Levandowsky and Winter(1971)}}?><label>Levandowsky and Winter(1971)</label><?label Levandowsky.1971?><mixed-citation>
Levandowsky, M. and Winter, D.: Distance between sets, Nature, 234, 34–35,
1971.</mixed-citation></ref>
      <ref id="bib1.bibx29"><?xmltex \def\ref@label{{Mayer et~al.(2023)Mayer, Techel, Schweizer, and van
Herwijnen}}?><label>Mayer et al.(2023)Mayer, Techel, Schweizer, and van
Herwijnen</label><?label mayer.2023?><mixed-citation>Mayer, S. I., Techel, F., Schw<?pagebreak page2914?>eizer, J., and van Herwijnen, A.: Prediction of natural dry-snow avalanche activity using physics-based snowpack simulations, EGUsphere [preprint], <ext-link xlink:href="https://doi.org/10.5194/egusphere-2023-646" ext-link-type="DOI">10.5194/egusphere-2023-646</ext-link>, 2023.</mixed-citation></ref>
      <ref id="bib1.bibx30"><?xmltex \def\ref@label{{McClung and Schaerer(1980)}}?><label>McClung and Schaerer(1980)</label><?label mcclung.1980?><mixed-citation>McClung, D. M. and Schaerer, P. A.: Snow avalanche size classification,
Avalanche Workshop, Vancouver, 3–5 November 1980, 12–27,
<uri>https://arc.lib.montana.edu/snow-science/objects/issw-1980-012-030.pdf</uri> (last access: 11 November 2022),
1980.</mixed-citation></ref>
      <ref id="bib1.bibx31"><?xmltex \def\ref@label{{McClung and Schaerer(1985)}}?><label>McClung and Schaerer(1985)</label><?label mcclung.1985?><mixed-citation>McClung, D. M. and Schaerer, P. A.: Characteristics of Flowing Snow and
Avalanche Impact Pressures, Ann. Glaciol., 6, 9–14,
<ext-link xlink:href="https://doi.org/10.3189/1985AoG6-1-9-14" ext-link-type="DOI">10.3189/1985AoG6-1-9-14</ext-link>, 1985.</mixed-citation></ref>
      <ref id="bib1.bibx32"><?xmltex \def\ref@label{{Molenaar(1998)}}?><label>Molenaar(1998)</label><?label molenaar.1998?><mixed-citation>
Molenaar, M.: An introduction to the theory of spatial object modelling for
GIS, Research Monographs in Geographic Information Systems, Taylor and
Francis, United Kingdom, ISBN 0-7484-0775-8, 1998.</mixed-citation></ref>
      <ref id="bib1.bibx33"><?xmltex \def\ref@label{{Moner et~al.(2013)Moner, Orgué, Gavaldà, and Bacardit}}?><label>Moner et al.(2013)Moner, Orgué, Gavaldà, and Bacardit</label><?label moner.2013?><mixed-citation>
Moner, I., Orgué, S., Gavaldà, J., and Bacardit, M.: How big is big: the
results of the avalanche size classification survey, International Snow
Science Workshop ISSW, Grenoble, 7–11 October 2023, 2013.</mixed-citation></ref>
      <ref id="bib1.bibx34"><?xmltex \def\ref@label{{Paul et~al.(2013)Paul, Barrand, Baumann, Berthier, Bolch, Casey,
Frey, Joshi, Konovalov, Le~Bris, M{\"{o}}lg, Nosenko, Nuth, Pope, Racoviteanu,
Rastner, Raup, Scharrer, Steffen, and Winsvold}}?><label>Paul et al.(2013)Paul, Barrand, Baumann, Berthier, Bolch, Casey,
Frey, Joshi, Konovalov, Le Bris, Mölg, Nosenko, Nuth, Pope, Racoviteanu,
Rastner, Raup, Scharrer, Steffen, and Winsvold</label><?label paul.2013?><mixed-citation>Paul, F., Barrand, N. E., Baumann, S., Berthier, E., Bolch, T., Casey, K. A.,
Frey, H., Joshi, S. P., Konovalov, V., Le Bris, R., Mölg, N., Nosenko,
G., Nuth, C., Pope, A., Racoviteanu, A., Rastner, P., Raup, B., Scharrer, K.,
Steffen, S., and Winsvold, S. H.: On the accuracy of glacier outlines derived
from remote-sensing data, Ann. Glaciol., 54, 171–182,
<ext-link xlink:href="https://doi.org/10.3189/2013AoG63A296" ext-link-type="DOI">10.3189/2013AoG63A296</ext-link>, 2013.</mixed-citation></ref>
      <ref id="bib1.bibx35"><?xmltex \def\ref@label{{{R Core Team}(2021)}}?><label>R Core Team(2021)</label><?label R_programming?><mixed-citation>R Core Team: R: A Language and Environment for Statistical Computing, R
Foundation for Statistical Computing, Vienna, Austria,
<uri>https://www.R-project.org/</uri> (last access: 25 August 2023), 2021.</mixed-citation></ref>
      <ref id="bib1.bibx36"><?xmltex \def\ref@label{{Schweizer et~al.(2020)Schweizer, Mitterer, Techel, Stoffel, and
Reuter}}?><label>Schweizer et al.(2020)Schweizer, Mitterer, Techel, Stoffel, and
Reuter</label><?label schweizer.2020?><mixed-citation>Schweizer, J., Mitterer, C., Techel, F., Stoffel, A., and Reuter, B.: On the relation between avalanche occurrence and avalanche danger level, The Cryosphere, 14, 737–750, <ext-link xlink:href="https://doi.org/10.5194/tc-14-737-2020" ext-link-type="DOI">10.5194/tc-14-737-2020</ext-link>, 2020.</mixed-citation></ref>
      <ref id="bib1.bibx37"><?xmltex \def\ref@label{{Sovilla et~al.(2008)Sovilla, Schaer, and Rammer}}?><label>Sovilla et al.(2008)Sovilla, Schaer, and Rammer</label><?label sovilla.2008?><mixed-citation>Sovilla, B., Schaer, M., and Rammer, L.: Measurements and analysis of
full-scale avalanche impact pressure at the Vallée de la Sionne test site,
Cold Reg. Sci. Technol., 51, 122–137,
<ext-link xlink:href="https://doi.org/10.1016/j.coldregions.2007.05.006" ext-link-type="DOI">10.1016/j.coldregions.2007.05.006</ext-link>, 2008.</mixed-citation></ref>
      <ref id="bib1.bibx38"><?xmltex \def\ref@label{{Stewart(2001)}}?><label>Stewart(2001)</label><?label stewart.2001?><mixed-citation>Stewart, T. R.: Principles of forecasting: a handbook for researchers and
practitioners, chap. Improving reliability of judgemental forecasts,
Kluwer Academic Publisher, 81–106, <ext-link xlink:href="https://doi.org/10.1007/978-0-306-47630-3_5" ext-link-type="DOI">10.1007/978-0-306-47630-3_5</ext-link>, 2001.</mixed-citation></ref>
      <ref id="bib1.bibx39"><?xmltex \def\ref@label{{swisstopo(2020a)}}?><label>swisstopo(2020a)</label><?label swisstopo.2020a?><mixed-citation>swisstopo: SWISSIMAGE – Das digitale Orthofotomosaik der Schweiz,
<uri>https://www.swisstopo.admin.ch/de/geodata/images/ortho/swissimage10.html</uri> (last access: 16 February 2023),
2020a.</mixed-citation></ref>
      <ref id="bib1.bibx40"><?xmltex \def\ref@label{{swisstopo(2020b)}}?><label>swisstopo(2020b)</label><?label swisstopo.2020b?><mixed-citation>swisstopo: Swiss Map Raster- Produktdokumentation,
<uri>https://www.swisstopo.admin.ch/content/swisstopo-internet/de/swisstopo/documents.download/swisstopo-internet/de/documents/karto-documents/shop/SMRProduktdokumentation_D.pdf</uri> (last access: 16 February 2023), 2020b.
</mixed-citation></ref><?xmltex \hack{\newpage}?>
      <ref id="bib1.bibx41"><?xmltex \def\ref@label{{Techel(2020)}}?><label>Techel(2020)</label><?label tech20phdThesis?><mixed-citation>Techel, F.: On consistency and quality in public avalanche forecasting: a
data-driven approach to forecast verification and to refining definitions of
avalanche danger, Ph.D thesis, Department of Geography, University of
Zurich, Zurich Switzerland, <ext-link xlink:href="https://doi.org/10.13140/RG.2.2.23044.76163" ext-link-type="DOI">10.13140/RG.2.2.23044.76163</ext-link>, 2020.</mixed-citation></ref>
      <ref id="bib1.bibx42"><?xmltex \def\ref@label{{Techel et~al.(2022)Techel, Mayer, P\'{e}rez-Guill\'{e}n, Schmudlach, and
Winkler}}?><label>Techel et al.(2022)Techel, Mayer, Pérez-Guillén, Schmudlach, and
Winkler</label><?label techel.2022?><mixed-citation>Techel, F., Mayer, S., Pérez-Guillén, C., Schmudlach, G., and Winkler, K.: On the correlation between a sub-level qualifier refining the danger level with observations and models relating to the contributing factors of avalanche danger, Nat. Hazards Earth Syst. Sci., 22, 1911–1930, <ext-link xlink:href="https://doi.org/10.5194/nhess-22-1911-2022" ext-link-type="DOI">10.5194/nhess-22-1911-2022</ext-link>, 2022.</mixed-citation></ref>
      <ref id="bib1.bibx43"><?xmltex \def\ref@label{{{U.S. Department of Agriculture}(1961)}}?><label>U.S. Department of Agriculture(1961)</label><?label usfs.1961?><mixed-citation>U.S. Department of Agriculture: Snow avalanches: a handbook of forecasting
and control measures, Agricultural Handbook, 194, 84 pp.,
<uri>https://naldc.nal.usda.gov/download/CAT87208423/pdf</uri> (last access: 11 November 2022), 1961.</mixed-citation></ref>
      <ref id="bib1.bibx44"><?xmltex \def\ref@label{{{Van Coillie} et~al.(2014){Van Coillie}, Gardin, Anseel, Duyck,
Verbeke, and {De Wulf}}}?><label>Van Coillie et al.(2014)Van Coillie, Gardin, Anseel, Duyck,
Verbeke, and De Wulf</label><?label vancoillie.2014?><mixed-citation>Van Coillie, F. M., Gardin, S., Anseel, F., Duyck, W., Verbeke, L. P., and
De Wulf, R. R.: Variability of operator performance in remote-sensing image
interpretation: the importance of human and external factors, Int.
J. Remote Sens., 35, 754–778, <ext-link xlink:href="https://doi.org/10.1080/01431161.2013.873152" ext-link-type="DOI">10.1080/01431161.2013.873152</ext-link>,
2014.</mixed-citation></ref>
      <ref id="bib1.bibx45"><?xmltex \def\ref@label{{Vickers et~al.(2016)Vickers, Eckerstorfer, Malnes, Larsen, and
Hindberg}}?><label>Vickers et al.(2016)Vickers, Eckerstorfer, Malnes, Larsen, and
Hindberg</label><?label Vickers.2016?><mixed-citation>Vickers, H., Eckerstorfer, M., Malnes, E., Larsen, Y., and Hindberg, H.: A
method for automated snow avalanche debris detection through use of synthetic
aperture radar (SAR) imaging, Earth and Space Science, 3, 446–462,
<ext-link xlink:href="https://doi.org/10.1002/2016ea000168" ext-link-type="DOI">10.1002/2016ea000168</ext-link>, 2016.</mixed-citation></ref>
      <ref id="bib1.bibx46"><?xmltex \def\ref@label{{V\"{o}lk(2020)}}?><label>Völk(2020)</label><?label voelk2020?><mixed-citation>
Völk, M. S.: Analyse der Beziehung zwischen Lawinenauslösung und
prognostizierter Lawinengefahr: Quantitative Darstellung einer regionalen
Lawinenaktivität am Beispiel Davos (CH), Master thesis,
Leopold-Franzens-Universität Innsbruck, 2020.</mixed-citation></ref>
      <ref id="bib1.bibx47"><?xmltex \def\ref@label{{Weller and Mann(1997)}}?><label>Weller and Mann(1997)</label><?label weller.1997?><mixed-citation>Weller, S. and Mann, N.: Assessing rater performance without a “gold
standard” using consensus theory, Med. Decis. Making, 17, 71–79,
<ext-link xlink:href="https://doi.org/10.1177/0272989X9701700108" ext-link-type="DOI">10.1177/0272989X9701700108</ext-link>, 1997.</mixed-citation></ref>
      <ref id="bib1.bibx48"><?xmltex \def\ref@label{{Wever et~al.(2018)Wever, Vera~Valero, and Techel}}?><label>Wever et al.(2018)Wever, Vera Valero, and Techel</label><?label wever.2018?><mixed-citation>Wever, N., Vera Valero, C., and Techel, F.: Coupled snow cover and avalanche
dynamics simulations to evaluate wet snow avalanche activity, J.
Geophys. Res.-Earth, 123, 1772–1796,
<ext-link xlink:href="https://doi.org/10.1029/2017JF004515" ext-link-type="DOI">10.1029/2017JF004515</ext-link>, 2018.</mixed-citation></ref>
      <ref id="bib1.bibx49"><?xmltex \def\ref@label{{Wong et~al.(2022)Wong, Paritosh, and Bollacker}}?><label>Wong et al.(2022)Wong, Paritosh, and Bollacker</label><?label wong.2022?><mixed-citation>Wong, K., Paritosh, P., and Bollacker, K.: Are ground truth labels
reproducible? An empirical study, in: Proceedings of ML Evaluation
Standards Workshop at ICLR 2022, 25–29 April 2022, 1–12,
<uri>https://ml-eval.github.io/assets/pdf/GroundTruthReproducibilityICLRSubmitted.pdf</uri> (last access: 25 August 2023),
2022.</mixed-citation></ref>

  </ref-list></back>
    <!--<article-title-html>Avalanche size estimation and avalanche outline determination by experts: reliability and implications for practice</article-title-html>
<abstract-html/>
<ref-html id="bib1.bib1"><label>American Avalanche Association(2022)</label><mixed-citation>
      
American Avalanche Association: Snow, Weather and Avalanches: Observation
Guidelines for Avalanche Programs in the United States, 4th edn.,
ISBN 979-8-218-05765-7, 2022.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib2"><label>Ardizzone et al.(2002)Ardizzone, Cardinali, Carrara, Guzzetti, and
Reichenbach</label><mixed-citation>
      
Ardizzone, F., Cardinali, M., Carrara, A., Guzzetti, F., and Reichenbach, P.: Impact of mapping errors on the reliability of landslide hazard maps, Nat. Hazards Earth Syst. Sci., 2, 3–14, <a href="https://doi.org/10.5194/nhess-2-3-2002" target="_blank">https://doi.org/10.5194/nhess-2-3-2002</a>, 2002.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib3"><label>Bianchi et al.(2021)Bianchi, Grahn, Eckerstorfer, Malnes, and
Vickers</label><mixed-citation>
      
Bianchi, F. M., Grahn, J., Eckerstorfer, M., Malnes, E., and Vickers, H.: Snow
Avalanche Segmentation in SAR Images With Fully Convolutional Neural
Networks, IEEE J. Sel. Top. Appl., 14, 75–82, <a href="https://doi.org/10.1109/JSTARS.2020.3036914" target="_blank">https://doi.org/10.1109/JSTARS.2020.3036914</a>, 2021.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib4"><label>Birkeland and Greene(2011)</label><mixed-citation>
      
Birkeland, K. and Greene, E.: Accurately Assessing Avalanche Size: The Ins and
Outs of the R- and D- scales, The Avalanche Review, 29, 27/32,
<a href="https://avalanche.org/wp-content/uploads/2018/08/11_TAR_BirkelandGreene.pdf" target="_blank"/> (last access: 5 January 2023),
2011.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib5"><label>Bowler(2006)</label><mixed-citation>
      
Bowler, N. E.: Explicitly Accounting for Observation Error in Categorical
Verification of Forecasts, Mon. Weather Rev., 134, 1600–1606,
<a href="https://doi.org/10.1175/MWR3138.1" target="_blank">https://doi.org/10.1175/MWR3138.1</a>, 2006.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib6"><label>Bozzini et al.(2012)Bozzini, Conedera, and Krebs</label><mixed-citation>
      
Bozzini, C., Conedera, M., and Krebs, P.: A New Monoplotting Tool to Extract
Georeferenced Vector Data and Orthorectified Raster Data from Oblique
Non-Metric Photographs, International Journal of Heritage in the Digital Era,
1, 499–518, <a href="https://doi.org/10.1260/2047-4970.1.3.499" target="_blank">https://doi.org/10.1260/2047-4970.1.3.499</a>, 2012.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib7"><label>Bozzini et al.(2013)Bozzini, Conedera, and Krebs</label><mixed-citation>
      
Bozzini, C., Conedera, M., and Krebs, P.: A new tool for facilitating the
retrieval and recording of the place name cultural heritage, The
International Archives of the Photogrammetry, Remote Sensing and Spatial
Information Sciences, XL-5/W2, 115–118,
<a href="https://doi.org/10.5194/isprsarchives-XL-5-W2-115-2013" target="_blank">https://doi.org/10.5194/isprsarchives-XL-5-W2-115-2013</a>, 2013.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib8"><label>Brardinoni et al.(2019)Brardinoni, Scotti, Sailer, and
Mair</label><mixed-citation>
      
Brardinoni, F., Scotti, R., Sailer, R., and Mair, V.: Evaluating sources of
uncertainty and variability in rock glacier inventories, Earth Surf.
Proc. Land., 44, 2450–2466,
<a href="https://doi.org/10.1002/esp.4674" target="_blank">https://doi.org/10.1002/esp.4674</a>, 2019.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib9"><label>Bühler et al.(2019)Bühler, Hafner, Zweifel, Zesiger, and
Heisig</label><mixed-citation>
      
Bühler, Y., Hafner, E. D., Zweifel, B., Zesiger, M., and Heisig, H.: Where are the avalanches? Rapid SPOT6 satellite data acquisition to map an extreme avalanche period over the Swiss Alps, The Cryosphere, 13, 3225–3238, <a href="https://doi.org/10.5194/tc-13-3225-2019" target="_blank">https://doi.org/10.5194/tc-13-3225-2019</a>, 2019.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib10"><label>Bühler et al.(2022)Bühler, Bebi, Christen, Margreth, Stoffel,
Stoffel, Marty, Schmucki, Caviezel, Kühne, Wohlwend, and
Bartelt</label><mixed-citation>
      
Bühler, Y., Bebi, P., Christen, M., Margreth, S., Stoffel, L., Stoffel, A., Marty, C., Schmucki, G., Caviezel, A., Kühne, R., Wohlwend, S., and Bartelt, P.: Automated avalanche hazard indication mapping on a statewide scale, Nat. Hazards Earth Syst. Sci., 22, 1825–1843, <a href="https://doi.org/10.5194/nhess-22-1825-2022" target="_blank">https://doi.org/10.5194/nhess-22-1825-2022</a>, 2022.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib11"><label>CAA(2023)</label><mixed-citation>
      
CAA: Avalanche Size,
<a href="https://www.avalanche-center.org/Education/glossary/avalanche-size.php" target="_blank"/> (last access: 4 January 2023),
2023.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib12"><label>Canadian Avalanche Association(2016)</label><mixed-citation>
      
Canadian Avalanche Association: Observation guidelines and recording
standards for weather, snowpack and avalanches,
<a href="https://cdn.ymaws.com/www.avalancheassociation.ca/resource/resmgr/standards_docs/OGRS2016web.pdf" target="_blank"/> (last access: 25 August 2023),
2016.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib13"><label>Cronbach(1947)</label><mixed-citation>
      
Cronbach, L. J.: Test “reliability”: Its
meaning and determination, Psychometrika, 12, 1–16,
<a href="https://doi.org/10.1007/bf02289289" target="_blank">https://doi.org/10.1007/bf02289289</a>, 1947.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib14"><label>EAWS(2023)</label><mixed-citation>
      
EAWS: Standards: Avalanche Size,
<a href="https://www.avalanches.org/standards/avalanche-size/" target="_blank"/> (last
access: 8 May 2023), 2023.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib15"><label>Eckerstorfer et al.(2016)Eckerstorfer, Bühler, Frauenfelder, and
Malnes</label><mixed-citation>
      
Eckerstorfer, M., Bühler, Y., Frauenfelder, R., and Malnes, E.: Remote sensing
of snow avalanches: Recent advances, potential, and limitations, Cold Reg.
Sci. Technol., 121, 126–140,
<a href="https://doi.org/10.1016/j.coldregions.2015.11.001" target="_blank">https://doi.org/10.1016/j.coldregions.2015.11.001</a>, 2016.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib16"><label>Eckerstorfer et al.(2017)Eckerstorfer, Malnes, and
Müller</label><mixed-citation>
      
Eckerstorfer, M., Malnes, E., and Müller, K.: A complete snow avalanche
activity record from a Norwegian forecasting region using Sentinel-1
satellite-radar data, Cold Reg. Sci. Technol., 144, 39–51,
<a href="https://doi.org/10.1016/j.coldregions.2017.08.004" target="_blank">https://doi.org/10.1016/j.coldregions.2017.08.004</a>, 2017.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib17"><label>Galli et al.(2008)Galli, Ardizzone, Cardinali, Guzzetti, and
Reichenbach</label><mixed-citation>
      
Galli, M., Ardizzone, F., Cardinali, M., Guzzetti, F., and Reichenbach, P.:
Comparing landslide inventory maps, Geomorphology, 94, 268–289,
<a href="https://doi.org/10.1016/j.geomorph.2006.09.023" target="_blank">https://doi.org/10.1016/j.geomorph.2006.09.023</a>, 2008.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib18"><label>Goffin and Olson(2011)</label><mixed-citation>
      
Goffin, R. D. and Olson, J. M.: Is It All Relative?: Comparative Judgments and
the Possible Improvement of Self-Ratings and Ratings of Others, Perspect. Psychol. Sci., 6, 48–60, <a href="https://doi.org/10.1177/1745691610393521" target="_blank">https://doi.org/10.1177/1745691610393521</a>, 2011.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib19"><label>Hafner(2023)</label><mixed-citation>
      
Hafner, E. D.: Data reliability study: avalanche size estimates and outlines, Envidat [data set],
<a href="https://doi.org/10.16904/envidat.423" target="_blank">https://doi.org/10.16904/envidat.423</a>, 2023.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib20"><label>Hafner et al.(2021)Hafner, Techel, Leinss, and
Bühler</label><mixed-citation>
      
Hafner, E. D., Techel, F., Leinss, S., and Bühler, Y.: Mapping avalanches with satellites – evaluation of performance and completeness, The Cryosphere, 15, 983–1004, <a href="https://doi.org/10.5194/tc-15-983-2021" target="_blank">https://doi.org/10.5194/tc-15-983-2021</a>, 2021.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib21"><label>Hafner et al.(2022)Hafner, Barton, Daudt, Wegner, Schindler, and
Bühler</label><mixed-citation>
      
Hafner, E. D., Barton, P., Daudt, R. C., Wegner, J. D., Schindler, K., and Bühler, Y.: Automated avalanche mapping from SPOT 6/7 satellite imagery with deep learning: results, evaluation, potential and limitations, The Cryosphere, 16, 3517–3530, <a href="https://doi.org/10.5194/tc-16-3517-2022" target="_blank">https://doi.org/10.5194/tc-16-3517-2022</a>, 2022.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib22"><label>Hagafors and Brehmer(1983)</label><mixed-citation>
      
Hagafors, R. and Brehmer, B.: Does having to justify one's judgments change the
nature of the judgment process?, Organ. Behav. Hum.
Perf., 31, 223–232,
<a href="https://doi.org/10.1016/0030-5073(83)90122-8" target="_blank">https://doi.org/10.1016/0030-5073(83)90122-8</a>, 1983.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib23"><label>Hendrikx et al.(2005)Hendrikx, Owens, Carran, and
Carran</label><mixed-citation>
      
Hendrikx, J., Owens, I., Carran, W., and Carran, A.: Avalanche activity in an
extreme maritime climate: The application of classification trees for
forecasting, Cold Reg. Sci. Technol., 43, 104–116, 2005.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib24"><label>Jamieson et al.(2014)Jamieson, Beglinger, and Wilson</label><mixed-citation>
      
Jamieson, B., Beglinger, R., and Wilson, D.: Case study of a large snow
avalanche in the Selkirk Mountains and reflections on the Canadian size
classification, Geohazards 6, 6th Canadian GeoHazards Conference, Kingston,
ON, Canada, 15–18 June 2014, 2014.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib25"><label>Korzeniowska et al.(2017)Korzeniowska, Bühler, Marty, and
Korup</label><mixed-citation>
      
Korzeniowska, K., Bühler, Y., Marty, M., and Korup, O.: Regional snow-avalanche detection using object-based image analysis of near-infrared aerial imagery, Nat. Hazards Earth Syst. Sci., 17, 1823–1836, <a href="https://doi.org/10.5194/nhess-17-1823-2017" target="_blank">https://doi.org/10.5194/nhess-17-1823-2017</a>, 2017.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib26"><label>Lampert et al.(2016)Lampert, Stumpf, and Gancarski</label><mixed-citation>
      
Lampert, T., Stumpf, A., and Gancarski, P.: An empirical study into annotator
agreement, ground truth estimation, and algorithm evaluation, IEEE
T. Image Process., 25, 2557–2572, 2016.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib27"><label>Lato et al.(2012)Lato, Frauenfelder, and Bühler</label><mixed-citation>
      
Lato, M. J., Frauenfelder, R., and Bühler, Y.: Automated detection of snow avalanche deposits: segmentation and classification of optical remote sensing imagery, Nat. Hazards Earth Syst. Sci., 12, 2893–2906, <a href="https://doi.org/10.5194/nhess-12-2893-2012" target="_blank">https://doi.org/10.5194/nhess-12-2893-2012</a>, 2012.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib28"><label>Levandowsky and Winter(1971)</label><mixed-citation>
      
Levandowsky, M. and Winter, D.: Distance between sets, Nature, 234, 34–35,
1971.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib29"><label>Mayer et al.(2023)Mayer, Techel, Schweizer, and van
Herwijnen</label><mixed-citation>
      
Mayer, S. I., Techel, F., Schweizer, J., and van Herwijnen, A.: Prediction of natural dry-snow avalanche activity using physics-based snowpack simulations, EGUsphere [preprint], <a href="https://doi.org/10.5194/egusphere-2023-646" target="_blank">https://doi.org/10.5194/egusphere-2023-646</a>, 2023.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib30"><label>McClung and Schaerer(1980)</label><mixed-citation>
      
McClung, D. M. and Schaerer, P. A.: Snow avalanche size classification,
Avalanche Workshop, Vancouver, 3–5 November 1980, 12–27,
<a href="https://arc.lib.montana.edu/snow-science/objects/issw-1980-012-030.pdf" target="_blank"/> (last access: 11 November 2022),
1980.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib31"><label>McClung and Schaerer(1985)</label><mixed-citation>
      
McClung, D. M. and Schaerer, P. A.: Characteristics of Flowing Snow and
Avalanche Impact Pressures, Ann. Glaciol., 6, 9–14,
<a href="https://doi.org/10.3189/1985AoG6-1-9-14" target="_blank">https://doi.org/10.3189/1985AoG6-1-9-14</a>, 1985.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib32"><label>Molenaar(1998)</label><mixed-citation>
      
Molenaar, M.: An introduction to the theory of spatial object modelling for
GIS, Research Monographs in Geographic Information Systems, Taylor and
Francis, United Kingdom, ISBN 0-7484-0775-8, 1998.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib33"><label>Moner et al.(2013)Moner, Orgué, Gavaldà, and Bacardit</label><mixed-citation>
      
Moner, I., Orgué, S., Gavaldà, J., and Bacardit, M.: How big is big: the
results of the avalanche size classification survey, International Snow
Science Workshop ISSW, Grenoble, 7–11 October 2023, 2013.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib34"><label>Paul et al.(2013)Paul, Barrand, Baumann, Berthier, Bolch, Casey,
Frey, Joshi, Konovalov, Le Bris, Mölg, Nosenko, Nuth, Pope, Racoviteanu,
Rastner, Raup, Scharrer, Steffen, and Winsvold</label><mixed-citation>
      
Paul, F., Barrand, N. E., Baumann, S., Berthier, E., Bolch, T., Casey, K. A.,
Frey, H., Joshi, S. P., Konovalov, V., Le Bris, R., Mölg, N., Nosenko,
G., Nuth, C., Pope, A., Racoviteanu, A., Rastner, P., Raup, B., Scharrer, K.,
Steffen, S., and Winsvold, S. H.: On the accuracy of glacier outlines derived
from remote-sensing data, Ann. Glaciol., 54, 171–182,
<a href="https://doi.org/10.3189/2013AoG63A296" target="_blank">https://doi.org/10.3189/2013AoG63A296</a>, 2013.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib35"><label>R Core Team(2021)</label><mixed-citation>
      
R Core Team: R: A Language and Environment for Statistical Computing, R
Foundation for Statistical Computing, Vienna, Austria,
<a href="https://www.R-project.org/" target="_blank"/> (last access: 25 August 2023), 2021.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib36"><label>Schweizer et al.(2020)Schweizer, Mitterer, Techel, Stoffel, and
Reuter</label><mixed-citation>
      
Schweizer, J., Mitterer, C., Techel, F., Stoffel, A., and Reuter, B.: On the relation between avalanche occurrence and avalanche danger level, The Cryosphere, 14, 737–750, <a href="https://doi.org/10.5194/tc-14-737-2020" target="_blank">https://doi.org/10.5194/tc-14-737-2020</a>, 2020.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib37"><label>Sovilla et al.(2008)Sovilla, Schaer, and Rammer</label><mixed-citation>
      
Sovilla, B., Schaer, M., and Rammer, L.: Measurements and analysis of
full-scale avalanche impact pressure at the Vallée de la Sionne test site,
Cold Reg. Sci. Technol., 51, 122–137,
<a href="https://doi.org/10.1016/j.coldregions.2007.05.006" target="_blank">https://doi.org/10.1016/j.coldregions.2007.05.006</a>, 2008.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib38"><label>Stewart(2001)</label><mixed-citation>
      
Stewart, T. R.: Principles of forecasting: a handbook for researchers and
practitioners, chap. Improving reliability of judgemental forecasts,
Kluwer Academic Publisher, 81–106, <a href="https://doi.org/10.1007/978-0-306-47630-3_5" target="_blank">https://doi.org/10.1007/978-0-306-47630-3_5</a>, 2001.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib39"><label>swisstopo(2020a)</label><mixed-citation>
      
swisstopo: SWISSIMAGE – Das digitale Orthofotomosaik der Schweiz,
<a href="https://www.swisstopo.admin.ch/de/geodata/images/ortho/swissimage10.html" target="_blank"/> (last access: 16 February 2023),
2020a.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib40"><label>swisstopo(2020b)</label><mixed-citation>
      
swisstopo: Swiss Map Raster- Produktdokumentation,
<a href="https://www.swisstopo.admin.ch/content/swisstopo-internet/de/swisstopo/documents.download/swisstopo-internet/de/documents/karto-documents/shop/SMRProduktdokumentation_D.pdf" target="_blank"/> (last access: 16 February 2023), 2020b.


    </mixed-citation></ref-html>
<ref-html id="bib1.bib41"><label>Techel(2020)</label><mixed-citation>
      
Techel, F.: On consistency and quality in public avalanche forecasting: a
data-driven approach to forecast verification and to refining definitions of
avalanche danger, Ph.D thesis, Department of Geography, University of
Zurich, Zurich Switzerland, <a href="https://doi.org/10.13140/RG.2.2.23044.76163" target="_blank">https://doi.org/10.13140/RG.2.2.23044.76163</a>, 2020.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib42"><label>Techel et al.(2022)Techel, Mayer, Pérez-Guillén, Schmudlach, and
Winkler</label><mixed-citation>
      
Techel, F., Mayer, S., Pérez-Guillén, C., Schmudlach, G., and Winkler, K.: On the correlation between a sub-level qualifier refining the danger level with observations and models relating to the contributing factors of avalanche danger, Nat. Hazards Earth Syst. Sci., 22, 1911–1930, <a href="https://doi.org/10.5194/nhess-22-1911-2022" target="_blank">https://doi.org/10.5194/nhess-22-1911-2022</a>, 2022.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib43"><label>U.S. Department of Agriculture(1961)</label><mixed-citation>
      
U.S. Department of Agriculture: Snow avalanches: a handbook of forecasting
and control measures, Agricultural Handbook, 194, 84 pp.,
<a href="https://naldc.nal.usda.gov/download/CAT87208423/pdf" target="_blank"/> (last access: 11 November 2022), 1961.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib44"><label>Van Coillie et al.(2014)Van Coillie, Gardin, Anseel, Duyck,
Verbeke, and De Wulf</label><mixed-citation>
      
Van Coillie, F. M., Gardin, S., Anseel, F., Duyck, W., Verbeke, L. P., and
De Wulf, R. R.: Variability of operator performance in remote-sensing image
interpretation: the importance of human and external factors, Int.
J. Remote Sens., 35, 754–778, <a href="https://doi.org/10.1080/01431161.2013.873152" target="_blank">https://doi.org/10.1080/01431161.2013.873152</a>,
2014.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib45"><label>Vickers et al.(2016)Vickers, Eckerstorfer, Malnes, Larsen, and
Hindberg</label><mixed-citation>
      
Vickers, H., Eckerstorfer, M., Malnes, E., Larsen, Y., and Hindberg, H.: A
method for automated snow avalanche debris detection through use of synthetic
aperture radar (SAR) imaging, Earth and Space Science, 3, 446–462,
<a href="https://doi.org/10.1002/2016ea000168" target="_blank">https://doi.org/10.1002/2016ea000168</a>, 2016.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib46"><label>Völk(2020)</label><mixed-citation>
      
Völk, M. S.: Analyse der Beziehung zwischen Lawinenauslösung und
prognostizierter Lawinengefahr: Quantitative Darstellung einer regionalen
Lawinenaktivität am Beispiel Davos (CH), Master thesis,
Leopold-Franzens-Universität Innsbruck, 2020.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib47"><label>Weller and Mann(1997)</label><mixed-citation>
      
Weller, S. and Mann, N.: Assessing rater performance without a “gold
standard” using consensus theory, Med. Decis. Making, 17, 71–79,
<a href="https://doi.org/10.1177/0272989X9701700108" target="_blank">https://doi.org/10.1177/0272989X9701700108</a>, 1997.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib48"><label>Wever et al.(2018)Wever, Vera Valero, and Techel</label><mixed-citation>
      
Wever, N., Vera Valero, C., and Techel, F.: Coupled snow cover and avalanche
dynamics simulations to evaluate wet snow avalanche activity, J.
Geophys. Res.-Earth, 123, 1772–1796,
<a href="https://doi.org/10.1029/2017JF004515" target="_blank">https://doi.org/10.1029/2017JF004515</a>, 2018.

    </mixed-citation></ref-html>
<ref-html id="bib1.bib49"><label>Wong et al.(2022)Wong, Paritosh, and Bollacker</label><mixed-citation>
      
Wong, K., Paritosh, P., and Bollacker, K.: Are ground truth labels
reproducible? An empirical study, in: Proceedings of ML Evaluation
Standards Workshop at ICLR 2022, 25–29 April 2022, 1–12,
<a href="https://ml-eval.github.io/assets/pdf/GroundTruthReproducibilityICLRSubmitted.pdf" target="_blank"/> (last access: 25 August 2023),
2022.

    </mixed-citation></ref-html>--></article>
