Evaluating quality of data collected by volunteers for first level inspection of hydraulic structures in mountain catchments

Introduction Conclusions References


Introduction
There is an increasing interest in the use of citizen-based approaches to understand better the territory and hazard-related processes.To that end, there are different data collection approaches according to citizens' skills and time of involvement, e.g.crowdsourcing (Hudson-Smith et al., 2008), volunteered geographic information (Goodchild, 2007) or facilitated-volunteered geographic information (Seeger, 2008).Moreover, scientists have increasingly considered management approaches based upon the broader Figures concept of citizen science (Bonney et al., 2009).Thereby, volunteers are enlisted and trained according to survey and management needs (Devictor et al., 2010).
In disaster risk management, citizen science is linked to European and worldwide directives as the Hyogo Framework (European Commission, 2007;United Nations, 2005).Such directives promote citizen involvement to build a culture of resilience before, during and after a disaster strikes (European Commission, 2012).Therefore, modern approaches for emergency management promote exchange of information between local authorities and volunteer groups to support preparedness and preventive actions (Enders, 2001).From managers' perspective, there are substantial advantages of promoting citizen science projects.Opportunities not only stem from the increasing frequency, timeliness and coverage of surveillance activities but also on collecting useful information for decision-making (Flanaging and Metzger, 2008).However, citizenbased procedures should be tested and adapted according to quality requirements of data collection campaigns (Bordogna et al., 2014;EPA, 1997;Goodchild and Li, 2012;Gouveia and Fonseca, 2008).
Experiences of citizen science include data collection regarding water quality and biological aspects (Engel et al., 2002;Fore et al., 2001;Nicholson et al., 2002); forestry and ecosystem rehabilitation (Brandon et al., 2003;Gollan et al., 2012); biodiversity (Snäll, 2011); stream monitoring (Bjorkland et al., 2001;Yetman, 2002) and hydrological processes (Cifelli et al., 2005;Rinderer et al., 2012).Despite the variety of projects, limited research has been devoted to evaluate the effectiveness of citizen-based data, compared with professional methods (Danielsen et al., 2005).Furthermore, the data quality is the greatest challenge for the practical use of these data by scientists and decision makers (Riesch and Potter, 2014).There is a general lack of confidence due to the limited accuracy, non-comparability and completeness of citizen collected data (Conrad and Hilchey, 2011).
In spite of the challenges for citizen involvement, precision and completeness of data collected largely depends on the exhaustiveness of the inspection procedures (Galloway et al., 2006) the inspection campaigns.However, the extension of these trainings should consider available time, number and type of participants (Tweddle et al., 2012).To that end, Jordan et al. (2011) suggested that identification of technical data should be restricted while more general indicators can still be accurately obtained.These indicators may be quantitative and qualitative aspects that are easily recognizable from visual inspections (Gommerman and Monroe, 2012;Gouveia et al., 2004).Then, qualitative field methods are generally based on rating scales to report inspected conditions.Moreover, Bjorkland et al. (2001) proposed such citizen-based approaches as screening method without replacing more detailed procedures when they are needed.This study considers regular inspections with citizen-volunteer groups that are promoted by Civil Protection and local authorities of Friuli Venezia Giulia (FVG), Italy.In mountain areas, regular inspection of hydraulic structures is important due to their influence on water sediment processes.This is particularly evident on debris flow control for structures such as check dams.Evidence is also found in the potential aggravation of flood hazard due to obstruction and erosion of bridges and culverts (Mazzorana et al., 2010).Therefore, two inspection forms and a learning session were prepared to carry out first level inspections on the functional status of bridges and check dams.
Survey procedures aim at collecting standardized data to update regional databases concerning the functional status of hydraulic structures.Consequently, collected data should support decisions about obstructions or pre-screen potential problems for more technical and detailed inspections.
In this paper, we evaluate quality of data collected for first level inspection of bridges and check dams.Therefore, we address the following research questions: (1) how well were participants able to report on the functional status by distinguishing between available rating classes?(2) How effectively data were collected by volunteers as compared to the one collected by technicians?(3) How can survey procedures be improved?To that end, part 2 briefly describes the methodology applied for designing the forms and the data collection exercise.In part 3, we evaluated quality of collected data according to their accuracy, precision and completeness.Thereby, results are presented Introduction

Conclusions References
Tables Figures

Back Close
Full according to specific aspects for bridges and check dams, then into common aspects for both structures.In part 4, we discuss performance and feedback of participants.Finally, we highlight in the discussion and conclusions key points for the practical use of citizen-based data.

Design of the inspection forms and data collection exercise
Methods start with the design of two inspection forms for bridges and check dams.Data was collected during an exercise for the first level inspection of six structures, hereafter referred as inspection tests.The exercise comprise of a learning and a testing session.Both volunteers and technicians participated in the exercise to evaluate quality of data collected.However, volunteer groups included citizen-volunteers of Civil Protection, geosciences and social sciences students to account for differences in preliminary knowledge to fill the form.Participants were divided in a Control and Learning Group in order to identify potential improvements in the survey procedures.Some volunteers and technicians were part of the Control Group and carried out the inspection tests without attending the learning session.

Design of the inspection forms
Table 1 summarizes the forms' layout divided by sections.We defined the later with four risk managers of Civil Protection, Geological Survey and Forestry Service of FVG.Section I and III of the form identify the inspector and the inspection conditions.The inspection conditions comprise the level of accessibility, presence of stream water, occurrence of rainfall and snow.Simplified identification of the structure type and function was precompiled in section II together with the location, type, use and presence of connected structures, if available.Thereafter, section IV of the form refers to the functional status of the inspected structure.Functional status is the susceptibility or physical conditions of the structure that may affect the function type for which it was designed or Introduction

Conclusions References
Tables Figures

Back Close
Full built (Uzielli et al., 2008).Furthermore, the functional status is inspected by looking at three parameters according to the type of structure.
In case of bridges, parameter A focuses on the opening for the water flow and erosion of the pillar or abutments.Parameter B assesses levels of lateral obstruction, either at the structure location or at the stream channel.Component questions in parameter B require to inspect immediately upstream, downstream and at the structure location.We also included a question referring to the maximum free height of the structure.However, it was finally not considered due to safety limitations for citizen-volunteers when accessing the stream channel.For check dams, the focus of parameter A is on the status of the structure itself and downstream scouring.Parameter B distinguishes between consolidation and open check dams.Then, upstream obstruction is limited to the open check dam type.That distinction is due to the relevance of open check dams for retention of sediments, if there is a retention basin connected to the structure.Therefore, we included "Not apply" option for inspecting consolidation check dams.
In contrast, Parameter C addresses the same questions for bridges and check dams.
It refers to the worst condition while looking to the presence of protection works and erosion level at the stream banks.Then, we established a control distance of 20 m upstream and downstream of the structure.This distance was defined to reduce variability of assessments during the inspection.The 20 m allow inspectors to observe and to take pictures, even if accessibility to the structure is restricted.Section V of the form is to report the critical infrastructure within the same control distance.Finally, section VI distinguishes required actions to follow up the inspection based on the options provided in the form.The inspection forms adopted are available as supplementary material.
Overall, section IV of the form is the key section to report on the functional status.The other sections of the form become relevant to distinguish inspection conditions at different periods.Parameters in section IV comprises of a maximum of four questions (e.g.A1, A2, A3 and A4).Questions and options to report were defined according to inspection procedures for technicians regarding check dams (Comiti and dell' Agnese, 2010;von Maravic, 2010;Mazzorana, 2008;Province of British Columbia, 2000) and Introduction

Conclusions References
Tables Figures

Back Close
Full bridges (Burke Engineering, 1999;Ohio Department of Transportation, 2010;Servizio Forestale FVG, 2002).Despite available procedures for technicians, volunteers' involvement demand more structured and simpler forms to inspect the functional status.Similar to Yetman (2002), we used rating scales to standardize collected data while distinguishing minor problems from those concerns that are more serious.Then, we proposed three to five classes to rate the functional status, or simply two classes when reporting between presence and absence of the inspected aspects.Five classes were chosen when it was possible to give a range in precision while describing from minimum to maximum concerns.
In addition, all questions had alternative options to report unspecified answers such as "I don't know" and "Could not be answered".The latter represents conditions at the structure location (e.g.water level) that did not allow inspectors to provide an assessment.When the question itself did not specify the location to report, a multiple choice was included by specifying the problem's location: right, left or in correspondence with the structure.Finally, rating scales included visual schemes for guiding the inspector and highlighting potential problems (Burke Engineering, 1999;Jakob and Hungr, 2005;Provinzia Autonoma di Bolzano, 2006;Ohio Department of Transportation, 2010).

Data collection exercise in the municipality of Pontebba, Fella basin (Italy)
The exercise was carried out in the Fella River Basin, a mountain basin in the Eastern Italian Alps.Civil Protection selected the structures for the inspection tests of the Evaluation focused on the component questions for parameters A, B, C and Synthesis (i.e.section IV and VI in Table 1).First, we analyzed how well participants were able to report on the functional status by distinguishing between available rating classes in the inspection forms (see Supplement).For the data evaluation, we assigned ordinal scores between 1 and 5 to the rating scales for reporting on each component question.Score 1 represents the best condition whereas score 5 represents the worst condition.
According to the description coming along for each class, rating scales with three classes comprise the following set of ordinal scores: 1, 4 and 5, if options ranged between total absence, presence and presence beyond the control distance or inspected aspect, e.g.options for question C1 (Tables 3 and 4).Ordinal scores 1, 3 and 5, if options ranged between total absence, fair presence and fully presence of the inspected aspects, e.g.options for question B2 in case of check dams (Table 4).Instead, rating scales with two classes or yes/no questions were only assigned with ordinal scores 1 and 5.
In addition, unspecified answers (%Us) included classes "I don't know", "Could not be answered" or "No answer".We used %Us to distinguish when the inspector could not select any option between the rating scale.Tables 3 and 4 summarize results according to the component questions per parameter.Mean ordinal scores (X ) and standard deviations (S.D.) were calculated from the ratings that participants reported in Test-1, 2 and 3.
Then, we evaluated how effectively data were collected by V as compared to the one collected by T. Figures 2-6 summarize results according to the Pre-test, Tests-1, 2 and 3. Distinction was done between V and T within participants of the Learning and Control Groups.For that purpose, a frequency analysis was applied to the ordinal scores.
This consideration was based upon the relatively low sample, size and difference in number between the groups.We referred to the mode score for the data evaluation as it represents the class with the highest frequency.Consequently, we analyzed the devi-Introduction

Conclusions References
Tables Figures

Back Close
Full ation relative to the mode for each question (i.e.mode-off by one level).In addition, we used the following criteria to assess the quality of collected data (EPA, 1997, p. 19-20): -Accuracy, "degree of agreement between the data collected and the true value on the condition being measured".Then, we referred to "true value" as the mode score for T in Learning Group.Figures 2-6 aggregate the relative frequencies in four frequency classes with reference to the "true value" : equal or larger than 90 %, 70-90 %, 50-70 % and smaller than 50 %.We chose this aggregation to distinguish different accuracy levels for each group.In addition, we assumed agreement among group members when a question had a relative frequency of at least 70 %.Then, the overall agreement per parameter was calculated by the ratio "Question/Parameter", i.e. the number of questions with frequencies of at least 70 % between total questions per parameter.
-Precision, "refers to how well data collected are able to reproduce the result on the same group".For all participants, we represented precision by using the standard deviation (S.D.) in Tables 3 and 4. Instead, in Figs.2-6 we compare each group while looking at the mode scores and the mode-off by one level.The mode-off by one level is a range in precision given by generalizing rating scales from their ordinal scores.For example, we generalize rating scales from five to three classes by grouping: very low-to-low concerns, medium concerns and high-to-very-high concerns.Those are ordinal scores 1 and 2 on one side; scores 4 and 5 on the other one.Consequently, mode-off by one level in Figs.2-6 only distinguished questions where the scale generalization brought forth increments to the relative frequencies.
-Completeness, "measure of the amount of valid data actually obtained vs. the amount expected to be obtained".In Tables 3 and 4, completeness is evaluated by the amount of answers obtained between the rating scales as compared to the selection of unspecified answers.In Figs.2-6, we evaluated completeness by distinguishing questions with relative frequencies larger than 14 % in the options: Introduction

Conclusions References
Tables Figures

Back Close
Full "I don't know", "Could not be answered" and "No answer".We chose a threshold of 14 % to highlight questions with the lower completeness.It corresponds approximately to an absolute frequency of one participant in the Control Group or two participants in the Learning Group.
Other criteria such as comparability and representativeness were only considered in designing the form."Comparability represents how well data from one form can be compared to data from another"."Representativeness is the degree to which collected data actually represent the structure being inspected" EPA (1997, p. 19-20).Then, we referred to comparability by using a standard form for bridges and check dams.For representativeness, we required the inspector a photo record to support their choices and to provide additional information for the later examination of inspections.

Functional status of bridges for A and B parameters
Table 3 shows that A and B parameters in Test-1 have mean scores between 1 and 2 in the functional status.Lower ordinal scores represent the best condition for inspected aspects (Fig. 3a).However, it is worth to mention the lower complexity on the inspection test.Bridge-1 has minimum concerns on the functional status and no connected elements.Despite T in the Control Group, Fig. 2 presents overall agreement near to one for A parameter.That is the ratio Question/Parameter for A parameter and Test-1.For B parameter, overall agreement was reached only in the mode-off by one level.That represents lower precision in the B ratings indistinctly of the groups.
However, performance in Test-1 contrasts with the one in Test-2.Inspection complexity of Bridge-2 was higher due to stream water flowing along the structure's pillars and abutments (Fig. 3b).Table 3 highlights for A parameter higher frequency of unspecified answers and ordinal scores with standard deviations larger than one.For A Parameter, Fig. 2 shows accuracy levels below the relative frequency of 70 %.Consequently, there is disagreement in the mode score between V and T, indistinctly of the groups.The presence of erosion in Test-2, i.e.Question A1, was mostly rated by participants as Introduction

Conclusions References
Tables Figures

Back Close
Full "Could not be answered", "No answer" or "No erosion".Moreover, those who reported erosion in the pillar and abutment did not distinguish among erosion presence with or without the stream water along the basis.
For A parameter in Test-3, Fig. 2 shows better performance for T and V of the Learning Group as compared to the Control Group.Difference in performance could represent some influence of the learning session.However, it also denotes the need for adjusting questions to avoid misunderstandings.That is the case of question A3, which should explicitly address the status of protection works for downstream scouring in bridges (Fig. 3c).For B parameter, T and V only reached accuracy levels above 70 % when looking at the mode-off by one level, indistinctly of the test.However, question B3 (Presence of islands with shrubs or man-made structures that reduce the opening for the flow) had the lowest precision in Table 3 and Fig. 2.Then, question B3 should be split for better distinguishing presence of islands with vegetation from man-made obstructions.

Functional status of check dams for A and B parameters
In Table 4, Check dam-1 has the worst functional status for A parameter among all inspected check dams.In Test-1 and question A4, ratings reported by all participants have mean scores of more than 4. Instead, for parameter B, Test-2 has mean scores of more than 4. In addition, Table 4 shows larger standard deviations for Test-1 and 2. Despite the functional status, the presence of connected elements to these structures also contributed to the larger standard deviations.Thus, complexity in the inspection was higher due to the presence of a secondary structure in Check dam-1 and a retention basin in Check dam-2 (Fig. 3d and e).
Figure 4 shows the lower accuracy levels and overall agreement ratio for A parameter in Test-3.Then, question A1 was the least accurate for V in Test-2 and Test-3.
In addition, Table 4 further highlights lower accuracy levels for question A1 with standard deviations above one.Those results may be explained on the rating scale we used, (see Supplement for check dams).For question A1, rating classes did not distin-Introduction

Conclusions References
Tables Figures

Back Close
Full guish slight deviations from the strong ones.Then, medium concerns were not explicitly within the available options.This fact is also evident in Table 4 from the ordinal scores that we assigned (1, 4 and 5).
In addition, Fig. 4 shows higher frequencies of unspecified answers for question A2 and A3 in Test-1 and Test-2.In Test-1, unspecified answers were due to the water level at the basis of the structure."Could not be answered" was even the preferred option by T in the Learning Group.In Test-2, visibility at the basis of the structure was limited due to the sediment accumulation.Finally, question A4 denoted higher frequencies of unspecified answers for all structures (Table 4 and Fig. 4).Description of question A4 should be reviewed to avoid misunderstanding with question A2.That is the case of connected structures for protection of downstream scouring.Classes to report in question A4 should be extended to consider all possible functional conditions.
For B parameter, questions B1 and B2 have the lowest completeness in Pre-Test and Test-1.Those questions were not relevant for the consolidation check dam.Despite the "Does not apply" option, V and T in the Learning Groups still preferred not to answer.
Overall, T in the Learning Group were more precise than volunteer groups.T in the Control Group and volunteer groups improved their accuracy levels in the mode-off by one level.That is aggregating frequencies for ordinal scores 4-5 and 1-2 together in the B component questions.

Common aspects for the functional status: C parameter and synthesis
In questions C1 and C2 for bridges and check dams, participants only distinguished the upstream and downstream location from the field inspection (Test-1).That is comparing to the Pre-test results, which was the preliminary inspection test to the learning session.For the Pre-tests, mode scores were only assessable during the field inspections (Test-1) due to difficulty of participants to compile the form in front of a poster (Pre-Test).
Similar performance of the Pre-tests holds for Check dam-1 and Bridge-1, indistinctly of the parameter inspected (Figs.2-6).Introduction

Conclusions References
Tables Figures

Back Close
Full Overall agreement for C parameter was mostly reached when looking at the mode-off by one level.Thus, some adjustments in the rating classes are still needed.Description for C1 should refer to the length of protection works within the 20 m, instead of simply referring to the presence of protection works.For C2, high erosion level should not be related to absence of vegetation in the stream bank.Lack of vegetation may not only refer to erosion, it also depends of the stream bank material.For example, Check dam-1 were made of natural rock in the left stream banks and Check dam-2 has an upstream retention basin with bare banks due to frequent deposition of sediments.
Finally, the last part of the inspection form refers to the synthesis of the inspection.In Figs. 5 and 6, performance of T was clearly better.V in Control Group had the best performance among volunteer groups.However, T in Control and Learning groups did not agree for all structures.Then, participants still require a short handout portable to the field to support their inspections.

Performance and feedback of participants
Starting from responses to the registration questionnaire, we evaluated preliminary knowledge of participants for the inspection of hydraulic structures.To that end, we calculated relative frequencies of right responses to the questions for preliminary knowledge.Each group of participants was characterized using the average and the standard deviation (S.D.) between the maximum and minimum frequencies of right responses.
Results confirmed similar experience and preliminary knowledge of technician groups.In contrast, V in the Learning Group had higher variance on the frequencies of right responses.That is average frequency of 62.1 % and S.D. 11.9 %.In contrast, V in the Control Group had average frequency of 62.Then, we evaluated the performance in the inspection tests to adjust component questions.To that end, we used participants' preference for unspecified answers.However, we also considered disagreement among technicians of Learning and Control groups.For T in the Learning Group, accuracy levels on their inspection tests were generally better than the Control Group.That may be explained from the opportunities to discuss the form with the Learning group.In contrast, Control Group relied more on their preliminary knowledge and own experience for form compiling.Consequently, the inspection form may have different interpretation for T in Control Group.
For V, there was no major difference between Control and Learning Group.That could denote needs for an extended learning session, e.g. a discussion after every inspection test to clarify misunderstandings.However, the similarity in the performance could also refer to participating geosciences students in the Control Group.Their performance had lower dispersion as compared to the V Learning Group.Geosciences students generally have limited experience for the inspection of hydraulic structures, which may also limit different interpretations of the inspection form.
Finally, we used comments provided by participants during the sessions and the comments provided in the feedback form to initiate corrective actions (Table 5).Despite the needs for improvement, several comments proved the utility of the activity: "as a good initiative to instruct volunteers on the observation of the territory with preventive scope", "It joined theory and practice together on the field", "It helped to understand and inspect the functionality of the structure".

Discussion
First level inspection of hydraulic structures is a citizen science project to collect data on the functional status of bridges and check dams.Citizen involvement aims at supporting cleaning and pre-screening of potential problems concerning the structures' status.Civil Protection and technical services could use data collected by volunteers at different periods to update regional databases of existent structures.The additional Introduction

Conclusions References
Tables Figures

Back Close
Full value is on the increasing frequency, timeliness and coverage of surveillance activities.
That consideration especially holds in mountain catchments with a large number of hydraulic structures or where financial and human resources are limited (Danielsen et al., 2005;Holub and Hubl, 2008;De Jong, 2013).
In this research, we approached key aspects for evaluating quality of volunteers' data starting from the technical perspective.First, we assigned ordinal scores to the ratings in order to get an indication on the functional status of the inspected structures.The scores were defined according to descriptions coming along to the rating classes.Thus, we calculated mean scores and standard deviations from the ratings that participants reported for every structure.In addition, score's variability and participants' preference for unspecified answers helped to identify potential improvements in the inspection form.However, citizen science procedures require iterative design and testing to reduce uncertainty and misunderstandings.Visual inspections are in general more subjective than quantitative techniques to various sources of bias, which decrease accuracy and completeness of volunteer data (Gouveia and Fonseca, 2008).Therefore, it was useful to include unspecified options distinguishing limitations such as water level and inspection conditions, which are also complementary information to analyze the reports.From the technical perspective, citizen involvement requires simple but standardized forms coupled with quality assurance methods (Crall et al., 2011;Riesch and Potter, 2014).Thus, technicians could compare and use collected data for later examination.From the social perspective, a cornerstone beyond our research scope is the increasing volunteers' awareness of the water-sediment processes being addressed (Couvet et al., 2008).Consequently, citizen-based approaches to be fully effective require the combination of two practical aspects.These are recruiting and training strategies but also supporting information systems and mobile devices to facilitate data collection, access and validation (Newman et al., 2012).
To answer our second research question (effectiveness of citizen-based data), we made distinction among volunteer data with those obtained by professional staff.Pre-Introduction

Conclusions References
Tables Figures

Back Close
Full liminary knowledge of participants were assessed through responses to the registration questionnaire for the data collection exercise.Some participants were part of the Control Group without attending the Learning session.Thus, we used the mode score of technicians in Learning Group to distinguish different accuracy levels between volunteers and technicians in the two groups.
Differences in accuracy can be explained from the individual experience and knowledge of participants regarding the inspection of hydraulic structures.For instance, procedures may be differently interpreted from the larger experience of technicians when comparing among Learning and Control groups of technicians.Similarly, the preliminary knowledge of geosciences students may also influence their performance in the Control Group as compared to the volunteers in the Learning Group, mostly composed of citizen-volunteers of Civil Protection.However, differences between learning and control groups can also be evidences of the lack of training and unfamiliarity with the survey protocols.Then, glossary of terms and short handout is still required to support volunteer inspections in the field.
Compared with data collected by technicians, results from volunteers often have higher variance.We found that the use of rating scales with a range in precision of one level could cope with some variance in the volunteers' judgment.Previous studies indicated the advantages of rating classes when it is possible to describe the different range of concerns (Yetman, 2002).In addition, Rinderer et al. (2012) argues that detail scales leave the option to combine later the classes depending on the questions to be answered.
Finally, we estimated a ratio for overall agreement per parameter to identify the most subjective parameter.That is the ratio of total questions with frequencies of at least 70 % in the mode score between the total questions per parameter.We found that the most subjective parameters were the level of erosion at the stream bank (C parameter) and synthesis of the inspection.For the first one, the description coming along with the rating classes should be improved.While, for the latter one, the synthesis should be optional for volunteer inspectors.For other parameters, presence of connected elements

Conclusions References
Tables Figures

Back Close
Full to the surveyed structures increased the complexity of the inspection.For example, a secondary check dam downstream of the Check dam-1 and Bridge-2, and a retention basin in the Check dam-2.Then, extended training and field practice is still required to familiarize volunteers with those aspects.
Our last research question referred to potential improvements in survey procedures.
We used completeness of form compiling and feedback of participants to identify corrective actions.Rating scales should consider all possible functional conditions when distinction among different concerns is possible.In addition, component questions could still be extended for the status of the structure (A Parameter), if form for bridges should be further adapted for culverts.That is by considering component questions regarding inverts, embankments and the roadway condition at the location of the structure (Najafi and Bhattachar, 2011).Previous experiences have also emphasized the need for photo record to support later examination of volunteers data (Yetman, 2002), additional activity that we asked volunteers to perform.Regardless of their importance, not all volunteers took photographs in the field.Participants expressed difficulties to relate sequentially the photo record in the form.Thus, future exercises should consider advantages on the explosive growth in usage of mobile applications for smart phones.Volunteers and technicians could exploit such applications in the field for form compiling, completeness checking, data transferring and photo record.In addition, we could include other quality control methods in the mobile application such as embedded glossary, systematic tag and georeference of photographs.However, usability of such applications should have more research attention.That consideration is relevant to identify potential improvements in the quality of volunteers' data and to address the diversity of citizen volunteers getting involved (Newman et al., 2010).

Conclusions References
Tables Figures

Back Close
Full

Conclusions
Results showed that citizen volunteers could carry out first level inspections with comparable performance to technicians.Differences among the 11 technicians and 25 volunteers does not have high statistical significance when distinction is done among Control and Learning groups.However, key points can still be extracted from this dataset.
Those considerations are relevant for the use of volunteers' data on the functional status of hydraulic structures.It may also provide some guidance to researchers and practitioners interested on citizen-based data: 1. Volunteers could carry out first level inspections with comparable performance to technicians but with a pre-required range in precision.However, survey procedures clear enough require iterative design and testing to avoid uncertainty and misunderstandings.
2. In spite of the need to standardize reports, unstructured data such as comments and pictures are still required by managers to validate completeness and precision of volunteers' data.Then, it is crucial the systematic tagging and referencing of that data.
3. Unspecified answers may persist according to the complexity of connected elements to the structure, and the unexpected conditions for the inspection.Rating classes should specify when water or sediment did not allow the assessment.However, other options should be limited to facilitate the later examination of data.
4. Volunteer ratings should be considered as first level assessment.Managers could combine these ratings to get indexes on the status of the structure at parameter level.However, an indication of the overall completeness per parameter would still be needed for the later examination of volunteer inspections.
5. The use of scores to convert volunteer ratings is important to get an indication of the functional status.Since the rating scales are expressed in linguistic terms, Full ratings could be converted into numbers by using fuzzy set theory instead of ordinal scores.Conversion of volunteer data using scales of fuzzy terms could handle the pre-required ranges in precision (e.g. from low-to-very low).
Important considerations to improve and promote citizen science projects are related first, to limitations on citizen involvement due to different culture of volunteer activities and interest to participate.Second, training is relevant for the performance of volunteers but also for increasing awareness and preparedness on the causes and consequences of hydro meteorological hazards (Enders, 2001).For the first one, students are an alternative approach for citizens' recruitment where there is limited culture of volunteer activities.Universities could involve students of geosciences or social sciences to gain practical knowledge or better understanding their territory (Savan et al., 2003).For the latter one, future research should test the effectiveness of the learning session according to differences in preliminary knowledge of participants.However, replication exercises are still needed to improve the consistency and robustness of the data evaluation here presented, after initiating the corrective actions in survey procedures.
The Supplement related to this article is available online at doi:10.5194/nhessd-2-3577-2014-supplement.-Requires routine cleaning of blockages by hand C) Presence of protection works and with a maximum group of 10 volunteers.erosion level at the stream bank.
-Requires cleaning with support of equipment.
-Second level inspection.Other action than cleaning is required.V. Presence of anthropic elements.
Refer critical infrastructure next to the structure.VI.Synthesis of the inspection.
Provide general recommendation from available options.Introduction

Conclusions References
Tables Figures

Back Close
Full  -Protection works for scouring should be inspected not only for check dams but also for bridges when it applies.
-For bridges, obstructions in the floodplain should be also reported.
-Upstream obstruction should be reported for open and consolidation check dams.
-Rating classes and schemes will be adapted according to participants' comments and recommendations from the results section.
-Brief guidelines and glossary must be provided together with the inspection form.
Did you find useful the options provided in the form to answer the questions?
-When possible, rating scales with three or two classes should be extended to rate all possible status.
-Presence of human infrastructure should be open question to report other infrastructure besides roads and buildings.
-To avoid misunderstandings, the question regarding the presence of protection works will better refer to their length within the control distance.
-The form will emphasize to report the infrastructure that may be affected in case of high water levels.
Which aspects you did not like from the activity?
-All structures to inspect should have available information for the function type.
-Participants with technical background considered the indoors session long while citizen-volunteers requested more time to better understand the theory and to carry out the inspections.
-The inspection in front of the poster could be better used after both theory and practice have been explained.
-Information regarding the type of structure will be always precompiled in the form.
-The learning should start directly with the outdoors session and finishes with the indoors session.
-The indoors session will be carried out separately for each group of participants.Interaction between groups will be limited to the outdoors session.Introduction

Conclusions References
Tables Figures

Back Close
Full Discussion Paper | Discussion Paper | Discussion Paper | . Therefore, training activities are often required before starting Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Learning and Testing session.Structures are located in the municipality of Pontebba (Fig.1).The complexity of the inspection tests was different according to the functional status of the structures.Then, structures for the inspection tests accounted minimum and serious concerns for the assessment of A, B and C parameters.Structures also included connected elements, such as retention basin and secondary check dams for scouring protection.Table2 describesthe organization of the exercise divided by sessions and inspection tests.The exercise included an indoors and outdoors learning session.The aim of the Discussion Paper | Discussion Paper | Discussion Paper | learning session was to minimize number of unspecified answers.Then, the Learning Group attended the learning sessions before carrying out the inspection tests.Instead, the Control Group had one-day program and initiated inspection tests directly in the field after a common introductory session.The exercise was carried out with 36 participants up to two-days of involvement.Registration was made in the website of the activity by filling a questionnaire and selecting among one or two-days session according to participants' availability.Participants could register until the exercise day, which took place on May 2013.The registration aimed at identifying participants and investigating their preliminary knowledge on the inspection of hydraulic structures.It comprised of statements for true/false selection and closed questions for single selection.We assessed preliminary knowledge by asking participants about: causes and protective actions in case of debris flow; function of check dams; and potential consequences of blockages in culverts.See the website of the activity 1 for more details on the questionnaire and training material.At the end of the activity, participants provided feedback and submitted the pictures they took during the inspection tests, if any.Finally, we distinguished participants between Volunteers (V) and Technicians (T).Six technicians joined T Control Group and five in the T Learning Group.Volunteers included citizen-volunteers of Civil Protection, geosciences and social sciences students.However, two students of social sciences joined the data collection exercise and they only participated in the V Learning group.Students of geosciences were only available for the inspection tests during the first day.They were equally divided in the V Learning and Control groups.Then, 18 volunteers joined V Learning Group whereas seven volunteers were in the V Control Group.Most citizen-volunteers of Civil Protection were present in the V Learning Group as they are the target group for the inspection campaigns promoted by Civil Protection of FVG. Discussion Paper | Discussion Paper | Discussion Paper | 3 Evaluation on the quality of collected data by volunteers Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | 0 % and S.D. of 23.0 %.Difference on standard deviations for right responses denotes the diverse composition among volunteer groups.Four of seven members in the Control Group were students of geosciences.Instead, Learning Group was mostly comprised of citizen-volunteers of Civil Protection.Only four out of these 18 members were students of geosciences.Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper |

Acknowledgements.
This research was conducted in collaboration with the Civil Protection, technical services and local authorities of Friuli-Venezia-Giulia Region.Students of the Università degli Studi di Trieste were also participants of the research.Authors would like to thank Roxana Ciurean and Chiara Calligaris for their valuable support during the fieldwork period in terms of organizational aspects.We would also like to thank reviewers for their comments that help to improve the manuscript.This work is part of the Marie Curie ITN Changing Hydrometeorological Risks as Analyzed by a New Generation of European Scientists (CHANGES project), which is funded by the European Community's 7th Framework Programme FP7/2007-2013 under Grant Agreement No. 263953.Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper |

Figure 1 .Figure 2 .
Figure 1.Map indicating the location of the structures for the inspection tests.

Figure 4 .Figure 5 .
Figure 4. Relative frequencies for V and T of Leaning (LG) and Control Groups (CG).Parameter A and B in inspection tests for check dams.

Table 1 .
Form's layout for bridges and check dams.
II. Structure and function type Precompiled with data available from regional databases of hydraulic structures.III.Inspection conditions Distinguish conditions of regular inspections from those to intensify surveillance.IV.Functional status: Distinguish among the following possible actions: A) Condition of the structure; -No action is required.B) Level of obstruction at the structure;

Table 5 .
Feedback of participants to initiate corrective actions in survey procedures.