The experiment was realized on a cliff situated in Liverdun, a town close to Nancy in France. The test site has been chosen because of its history. It is an old quarry that became a climbing site, which has been closed after repetitive rockfalls. Moreover, it has been previously classified as presenting a high-level rockfall hazard (Moiriat et al., 2008). The methodology used to evaluate the rockfall hazard on the site is the following: (1) bibliographic review of all the documents available and (2) field recognitions. The first step leads to evaluate the risk area at a departmental scale. The second step leads to prioritize the level of rockfall hazard on the area of study. Therefore, the evaluation is made at a regional scale. The site is a 50 m long cliff of Jurassic limestone that is made up of massive blocks at the base and small blocks in the upper part (Delonca et al., 2013).

Three sectors were identified (Delonca et al., 2013):

Sector 1 presents a massive wall with well-defined stratification beds. A pluri-decametric fracture isolates a rock panel several hundreds of cubic meters in size. At the top, small unstable blocks are present (Fig. 1a).

Three populations were involved in the experiment.

A total of 38 first-year MSc students in the geosciences, confronted with a hazard assessment study for the first time. These students may later be asked to prepare hazard or risk maps, for example during an internship.

At first, an a priori assessment was requested from each population, without making use of any method. Then, two rockfall hazard assessment methods are used: the LPC method, which is a qualitative method frequently used in France, and a SMR-based method that relies on more quantitative information.

The on-site observations took place at the foot of the cliff. Each individual had a compass, a sclerometer and a geologist's hammer with him and had to report his observations and measurements on a separate form for each sector. On this form, each method was presented again, and all the used parameters were listed (cf. Fig. 3 for the LPC method).

Each individual had to make three types of rockfall hazard assessment: an a priori assessment, an assessment with the LPC method and an assessment with the SMR-based method. After the experiments were carried out, a descriptive statistical analysis was performed to compare the levels of the rockfall hazard assessment. Then, the results were analyzed using the chi-square independence test and the analysis of variance (ANOVA) method (Scheffé, 1959) to assess the significance of the influence of the level of expertise and of the influence of the method used in determining the rockfall hazard level. Finally, the means of the LPC and SMR-based method parameters were compared to identify those that most influence the rockfall hazard assessment.

An initial statistical treatment has been performed for the three sectors. Figure 5 and Table 8 show the results and exhibit the following.

Regarding the a priori assessment, there is a wide dispersion (standard deviation higher than 0.5) of the results in terms of rockfall hazard levels for all populations in all sectors, except for the experts in sector 3. Except for sector 3, where the populations give similar results, the students more often give a higher value for the rockfall hazard level, followed by the experts and finally the researchers. This can be explained by the level of inexperience of the students, who were initially impressed by the presence of cracks and overhanging blocks on the cliff.

The influence of the level of expertise and of the chosen method on the level of rockfall hazard could not be clearly determined from this preliminary descriptive analysis. This is the reason why statistical tests have also been performed.

To better understand the influence of the LPC and SMR-based methods on the level of rockfall hazard, a comparison of the mean values of the parameters used in these methods has been conducted (Table 14).

For each parameter presented, a two-by-two comparison of the mean values has been carried out. More precisely, we have compared the mean values of the levels given for each parameter by (i) the students and the researchers, (ii) the students and the experts and (iii) the researchers and the experts. The objective was to identify the mean values that are statistically similar and those that are different. This analysis could explain the influence of the chosen method previously highlighted, depending on the sector and the level of expertise.

The first step consists of checking the hypothesis of equal variance for each couple of the population. If the variances can be considered equals, it is then possible to compare the mean values. Let n1 and s12 be the size and the variance of the first population sample (students, researchers or experts), and let be n2 and s22 be the size and the variance of the second population sample. Under the hypothesis of equal population variances, the two estimates of the variance should be equal. This hypothesis has been tested (F test) and verified, and the comparison of the means can then be performed.

Let m1 and m2 be, respectively, the mean values of the first and second population samples under test. Both are unbiased estimates of the population means (μ1 and μ2). Under the assumption that μ1=μ2, we compute the tratio (Eq. 3), which is supposed to belong to a Student's t distribution with (n1+n2-2) degrees of freedom. This value has then to be compared to the interval [-tα/2tα/2], where α is the risk and (1-α) the confidence level: t=(m1-m2)/σ∗2(1n1)+(1/n2), where

m1 is the average of the first population tested (students, researchers or experts), and m2 is the average of the second population tested;

Table 16 summarizes the results and shows the following:

the predisposition to the instability (LPC method only) is the same, whatever the level of expertise and the sector;

Note that another analysis (principal component analysis) has been realized to compare directly the parameters used to evaluate the occurrence probability of the LPC method (Fig. 3). However, the analysis did not produce any concluding results.

During the experiment, three methods were used to assess the rockfall hazard. First, each participant was asked to assess the level of rockfall hazard from his own experience without using any tool, form or measurement. Second, a qualitative method (LPC method) and, third, a quantitative method (SMR-based method) were used. In both of these latter cases, a form was provided to guide the process of assessment.

We chose to use the LPC method due to its wide use in France for most official hazard mapping. This methodology aims to formalize the practice gradually developed over 2 decades through many field studies produced by both operational services of the French government and local authorities. Many experts contributed to the preparation of this guide, and some of these experts also participated in our experiment. The methodology proposed by the LPC method is somehow classical and follows the following steps as previously described:

a literature review, which avoids repeating studies or investigations already done;

Then, even though the LPC method is not an internationally recognized method, we may consider that it treats the hazard analysis in a common manner, similar to other methods. In contrast, the SMR method, also used in the present work, is commonly used in rock slope stability analysis along roadways (Budetta, 2004; Corominas Dulcet and Mavrouli, 2009; Irigaray et al., 2003; Tomas et al., 2012). It is derived from the globally used RMR method for the stability study of excavated slopes or underground excavations.

Even though it is a qualitative method, the LPC method helps the user to identify the block masses that are potentially instable and the volume concerned and leads to the assessment of the level of rockfall hazard in a similar manner by the three tested populations. However, the introduction of a paper form to assist in the process of the assessment of the level of rockfall hazard could have introduced a bias favoring the LPC method in terms of less dispersion in the assessment results when compared with the results of the SMR-based method. As a result, such a form could be promoted to engineers making use of the LPC method to enforce a common interpretation of some aspects of the method.

The SMR-based method shows results similar to those obtained by the LPC method for students. However, for researchers and experts, the dispersion in the results is more important. The assessment of the site activity is not precise enough to allow the same level of activity to be assessed by all the individuals. Surprisingly, the use of a partly quantitative method does not prevent differences in the final rockfall hazard level, mainly because of the assessment of the site activity, which is a subjective component of the method. The terms used to describe the activity may be clarified to reduce this effect. Moreover, the prevailing failure mechanism on the three sectors plays an important role in the use of the SMR-based method. Indeed, the SMR method is routinely used to assess rockfall susceptibility of failure mechanisms involving shear on joints (including flexural toppling) though it seems inapplicable to failures caused by traction. In the test site, the main failure mechanisms are column toppling, wedge slide and overhang failure. However, there also exists a failure mechanism that involves shearing of joints on the three studied sectors. Because it is not the main failure mechanism on the test site, the SMR-based method has been considered anyway into the analysis.

For the experiment, one test site was selected, among which three sectors could be defined. These three sectors provide repeatability in the analysis and produce more relevant results. However, because there are three sectors on the same site, all of them are composed of the same lithology. Hence, the results obtained in the present study only apply to this case study, and the experiment should be repeated on other sites with different lithology and geological structures to prove that the findings are generic. The test site was chosen because of its history, as it was a climbing site which has been closed after repetitive rockfalls. However, no quantitative information concerning the occurrence of the phenomenon was available. The site is not monitored, and the sole data considered in the analysis were the visual evidence of events. The scenario considered here is that an expert is in charge of the evaluation of a new site, and he does not have any temporal data available to him. Therefore, to propose a more complete analysis, it would be interesting to consider another test site where the estimation of temporal frequency is possible. A national French project is underway that may provide a framework to repeat this experiment on different study sites, with more students, researchers and experts involved, as well as other methods to be tested. However, no sites have yet been chosen to carry out this experiment again.

As explained previously, to carry out the statistical analysis, the three rockfall hazard levels were transformed into quantitative variables: the low-rockfall hazard level was coded to 1, the moderate-rockfall hazard level to 2 and the high-rockfall hazard level to 3, so as to reflect the increase of the rockfall hazard in a familiar and convenient way. However, to confirm that this coding has no influence on the results, we performed additional ANOVAs with different encodings. The following cases were used, respectively, for low-, moderate- and high-rockfall hazard levels:

0, 5 and 10;

Note that the quantification of the variable is used only for the statistical analysis.