Definition and impact of a quality index for radar-based reference measurements in the H-SAF precipitation product validation

The EUMETSAT Satellite Application Facility on Support to Operational Hydrology and Water Management (H-SAF) provides rainfall estimations based on infrared and microwave satellite sensors on board polar and geostationary satellites. The validation of these satellite estimations is performed by the H-SAF Precipitation Product Validation Group (PPVG). A common validation methodology has been defined inside the PPVG in order to make validation results from several institutes comparable and understandable. The validation of the PR-OBS-3 (blended infrared– microwave (IR–MW) instantaneous rainfall estimation) product using radar-based rainfall estimations as ground reference is described herein. A network of C-band and Kaband radars throughout Europe ensures a wide area coverage with different orographic configurations and climatological regimes, but the definition of a quality control protocol for obtaining consistent ground precipitation fields across several countries is required. Among the hydro-meteorological community, the evaluation of the data quality is a quite consolidated practice, even though a unique definition of a common evaluation methodology between different countries and institutions has not been set up yet. Inside H-SAF, the first definition of the quality index of the radar rainfall observations has been introduced at the Italian Civil Protection Department (DPC). In the evaluation of the DPC quality index, several parameters are considered, some measured by the radar itself (static clutter map, range distance, radial velocity, texture of differential reflectivity, texture of co-polar correlation coefficient and texture of differential phase shift) and some obtained by external sources (digital elevation model, freezing layer height). In some cases, corrections were applied for clutter and beam blocking. The DPC quality index was calculated and applied to some relevant meteorological events reported by a radar test site in Italy. The precipitation field derived by radar data was compared with the PR-OBS-3 precipitation product, with varying thresholds of quality index: the impact of the introduction of the quality index defined on the statistical results of the satellite product validation as well as their sensitivity to the threshold choice were thus evaluated. Results show that PRRMSE (a relative RMSE here introduced) is reduced from values between 2.5 and 3 to values around 1 when the quality threshold is increased from 0 (no threshold) to 0.8. Fractional standard error also decreases, from values around 2 to values around 1.5 in the same span of the quality threshold.

range distance, radial velocity, texture of differential reflectivity, texture of co-polar correlation coefficient and texture of differential phase shift) and some obtained by external sources (digital elevation model, freezing layer height).In some cases, corrections were applied for clutter and beam blocking.
The DPC quality index was calculated and applied to some relevant meteorological events reported by a radar test site in Italy.The precipitation field derived by radar data was compared with the PR-OBS-3 precipitation product, with varying thresholds of quality index: the impact of the introduction of the quality index defined on the statistical results of the satellite product validation as well as their sensitivity to the threshold choice were thus evaluated.Results show that PR-RMSE (a relative RMSE here introduced) is reduced from values between 2.5 and 3 to values around 1 when the quality threshold is increased from 0 (no threshold) to 0.8.Fractional standard error also decreases, from values around 2 to values around 1.5 in the same span of the quality threshold.

Introduction
The EUMETSAT Satellite Application Facility on Support to Operational Hydrology and Water Management (H-SAF) project started in 2005, with the objective of making available hydrological parameters estimations (instantaneous and cumulated rain rate, soil moisture at the surface and in the Published by Copernicus Publications on behalf of the European Geosciences Union. root zone, snow cover, quality and water equivalent) derived from satellite observations.The H-SAF project involves experts from 12 EUMETSAT member or cooperating states (Austria, Belgium, Bulgaria, Finland, France, Germany, Hungary, Italy, Poland, Romania, Slovakia and Turkey), and from the European Centre for Medium-Range Weather Forecasts (ECMWF).The H-SAF is hosted by the National Centre of Aeronautic Meteorology and Climatology (CNMCA) of the Italian National Meteorological Service.
The space-and time resolution of H-SAF satellite products is tuned to satisfy the needs of operational hydrology.Raw data from European and American satellites (Meteosat, NOAA, DMSP, NASA) are processed through specifically developed algorithms in order to obtain estimates of hydrometeorological parameters.A detailed presentation of the H-SAF programme and its associated activities, as well as of the satellite-based precipitation algorithms and products that have been developed so far, can be found in Mugnai et al. (2013).
In order to evaluate the performance of the products, it is very important to validate them by using independent datasets, i.e. data obtained by different sensors, and not involved in the product build-up.The independent validation is also necessary to verify the usefulness of the new products for monitoring and mitigation actions against floods, landslides and avalanches, and for evaluating water resources.For details about the H-SAF precipitation product validation, see Puca et al. (2013).
The previous works on satellite-based precipitation product validation (e.g.Ebert and Manton, 1998) adopt mainly the approach of re-mapping both satellite and ground data to the same reference grid.Inside the H-SAF, a different approach was chosen: to validate the satellite estimations in their native grid by remapping only the ground data.
In particular, when dealing with validation using radarbased estimations as reference data, these data come normally at a finer resolution than the one of satellite grids.Thus, radar maps have to be upscaled.It is important to introduce quality information for radar measurements in order to select the radar pixels to be involved in the upscaling or to weigh them on the basis of their liability.
The definition of an algorithm for radar quality calculation has been dealt with in recent years by different authors (e.g.Friedrich et al., 2006;Szturc et al., 2011).The algorithm here presented is based mainly on already assessed techniques, but introduces also some new approaches.
The present work is the first study focused on evaluating the impact on satellite precipitation estimations validation of the introduction of a quality filter on reference radar data before the upscaling stage.
The validation of the H-SAF precipitation product PR-OBS-3 using ground radar data and carried out by the H-SAF Precipitation Products Validation Group (PPVG) is here presented.The first section outlines the PR-OBS-3 satellite product, and the second gives an overview of the PPVG and its components, the framework of the radar data used for validation and the validation technique.
The theoretical treatment of the radar quality index, as well as the procedure derived from it and applied to data from an Italian C-band radar, is described in Sect. 4.
Finally, the impact of the introduction of this quality information on the validation of the PR-OBS-3 is evaluated in Sect. 5 for different quality thresholds.Conclusions are provided in Sect.6.

PR-OBS-3: precipitation rate estimation by GEO/IR data blended with LEO/MW measurements
The PR-OBS-3 is an instantaneous precipitation intensity product at the spatial and temporal resolution of the geostationary Spinning Enhanced Visible and InfraRed Imager (SEVIRI) sensor on board the Meteosat Second Generation (MSG) platform, generated by the rapid-update blended PMW-IR technique originally developed at the Naval Research Laboratory (Monterey, CA) by Turk et al. (2000).The combined use of IR brightness temperatures with the precipitation intensities from PMW allows for the scarcity of suitable PMW overpasses to be overcome, and provides an instantaneous precipitation intensity product with the spatial and temporal resolution of the GEO sensor on timescales consistent with the nature and development of the precipitating cloud systems.The geographic area covered by PR-OBS-3 is the whole H-SAF area (Puca et al., 2013), where the resolution degrades with increasing latitude.
The PR-OBS-3 product is based on the correlation, by using the statistical probability matching technique (Calheiros and Zawadzki, 1987) between the IR (10.8 µm) brightness temperatures measured by the SEVIRI sensor and PR-OBS-1and PR-OBS-2 (Casella et al., 2012;Sanò et al., 2013) products of ground precipitation rates derived from conical and cross-track PMW scanners, respectively.Look-up tables of geolocated relationships between the geostationary brightness temperatures and PMW precipitation rates are generated by considering space and time co-located observations, and updated every time new coincident PMW rainfall estimates and geostationary data are available.These relationships are based on the assignment of a rainfall intensity value to each SEVIRI pixel by taking into account that relationships older than 24 h with respect to the acquisition time of the processing IR SEVIRI brightness temperatures are considered unreliable, and consequently no rainfall intensity values are assigned until a refresh of the relationship is performed.
The calibration of brightness temperatures in terms of precipitation rate by means of MW measurements implies the existence of a good correlation between IR brightness temperatures and precipitation rate, which is fairly acceptable for convective precipitation, but less so for non-convective.Actually, in spite of the high spatial resolution and short refresh time, these kind of algorithms based on the IR data have often some difficulties in retrieving precipitation in the case of warm rain, or erroneously assign precipitation to cold, thin clouds such as cirrus.This is due to the fact that the blended technique assumes that the colder the cloud tops, the higher the associated precipitation intensity values.Nevertheless, the rapid update algorithm is currently the only operational algorithm allowing for precipitation rate estimates with the time resolution required for nowcasting.
The architecture of the PR-OBS-3 product generation chain is shown in Fig. 1.

Validation service for the H-SAF precipitation products
The PPVG is composed of experts from the national meteorological and hydrological institutes of Belgium, Bulgaria, Germany, Hungary, Italy, Poland, Slovakia and Turkey, under the coordination of the Italian Civil Protection Department (DPC).Hydrologists, meteorologists and precipitation ground data experts are involved in the product validation activities.A network of 4100 rain gauges and 54 meteorological radars provides reference ground data (Puca et al., 2013).
Since the beginning of the project, the importance of defining a common validation procedure has been clear in order to make the results obtained by several institutes comparable and clearly understandable.The main steps of this methodology have been identified inside the validation group, in collaboration with the product developers.The common validation methodology is based on ground data (radar, rain gauge and integrated) comparisons to produce large statistical (multi-categorical and continuous) and case study analysis.The products differ in data retrieval technique, space and time resolution such that every product needs a specific validation procedure tuned on its features.
In the case of the radar data used as ground reference, the main steps of the common validation methodology are as follows: -Ground data error analysis.
-Upscaling of radar data versus satellite native grid.
-Temporal comparison of precipitation products (satellite and ground).
Initially, every institute participating in the PPVG had developed its locally implemented validation software, following common guidelines.As the project progressed, the need for an improvement in the validation quality and consistency resulted in the definition of all the details of a unified validation software to be used by all the member institutes.

Quality of radar-based precipitation estimation
Quantitative precipitation estimation from ground-based weather radars is a cumbersome task considering it is influenced by several error sources (Wilson and Brandes, 1979).Even though some of these sources can be addressed to a reasonable extent, any quantitative use of radar rainfall products should take into account the quality of input radar data and related precipitation estimates.This is necessary both for radar data assimilation and validation of satellite-based precipitation products.
Starting from the paradigm that the quality is a subjective quantity, there is no unique way to determine it, nor is there a unique way to deal with the radar error sources.However, it is possible to provide a theoretical definition for data quality that might require specific setup for every radar system.
The quality is a random variable ranging between 0 and 1, which depends on the considered quality indicators (random variables related to the error sources).For each quality indicator, a relative quality index can be defined.The overall quality can then be computed as combination of the relative quality indices.
Assuming the radar systems are well maintained, the analysis will be focused on the following error sources: clutter, beam blocking, distance from the radar, height of measurement and attenuation.
As aforementioned, the present work is based on data from the DPC C-band radar named Il Monte (located in central Italy) at 700 m a.s.l., with significant orographic obstruction in the W-SW direction (see Fig. 2).

Ground clutter
The ground clutter can be evaluated using several methods, among which those employing only the Doppler information (ground clutter is expected to be basically stationary) might produce a suppression of precipitation echoes having the radial component of velocity close to zero.Consequently, any efficient clutter identification algorithm should also consider other information.A potential approach to discriminate the radar echoes generated by non-meteorological targets from weather returns relies on the combination of different quality indicators.Joss and Lee (1995) propose an approach based  2005) adopt a polarimetric approach, also for hydrometeor discrimination.The treatment here presented is a combination of the two approaches (Vulpiani et al., 2012) taking into account the following quality indicators: static clutter map (CMAP), radial velocity (V ), texture of differential reflectivity Z dr (TxZdr), texture of co-polar correlation coefficient ρ hv (TxRho) and texture of differential phase shift dp (TxPhi).CMAP is a volumetric map obtained by averaging a wide set of reflectivity data (expressed in mm 6 m −3 ) observed in clear-air conditions.It is worth noting that CMAP is dependent on the propagation conditions, so it would be recommended to build CMAP on a seasonal basis.
For each quality indicator X j (i.e.X 1 = CMAP, X 2 = V , X 3 = TxZdr, X 4 = TxRho, X 5 = TxPhi) the degree of membership to the non-meteorological target class d j is defined through a trapezoidal transformation function: where X i,j is the ith vertex of the trapezoid relative to the j th quality indicator.Table 1 shows the parameterization used for defining d j in the present work.With future adoption of a unified algorithm for quality index calculation by different countries, those parameters might be locally tuned, even though the use of the same parameter values by all validators would be preferable.The relative quality index q j associated to X j is then defined as the complementary of d j (i.e.q j = 1 − d j ): (2) Radar returns with associated low quality (i.e.q clutter < 0.6) can be finally rejected.No correction is applied.Figures 3-5 show an example of radar image, the corresponding Q clutter map and the clutter-filtered image.Some radar sites might not be able to measure polarimetric variables.An algorithm for clutter quality calculation suitable for such radars is presently in the course of implementation.

Beam blocking
In order to properly take into account the beam shielding effects an electromagnetic propagation model (EPM) can be used to identify the obstructed radial directions.The partial beam blockage (PBB) map, representing the occultation degree at a specific antenna elevation, can be retrieved by applying the simplified obstruction function proposed by Bech et al. (2003): where y is the difference between the height of the terrain and the height of the centre of the radar beam (h), and a is the radius of the beam cross section.The height of the centre Table 1.Parameters of the applied system for evaluating q clutter . of the radar beam h at a distance r can be written as (Doviak and Zrnić, 1993) where R is the Earth radius, θ the antenna elevation and H 0 the radar antenna height, and k e = 4/3 (assuming the wave propagation of the standard atmosphere).
The quality associated with the beam blocking can then be computed as the complementary of the PBB: (5) The estimated PBB might be compensated up to 0.7 as in Tabary (2007); consequently, the resulting quality would be (Fig. 6) As example, Fig. 7 shows the reflectitivity factor filtered for clutter and corrected for partial beam blocking.

Range distance
The quality of radar data decreases with increasing distance from the radar either for the beam broadening related to the spherical divergence of the electromagnetic waves or for the increasing height with respect to terrain (due to the Earth's curvature and the vertical variability of the refractive index of the atmosphere).Following the approach proposed by Friedrich et al. (2006), but introducing a square root, it can be evaluated as follows: for r ≤ r min r max −r r max −r min for r min < r < r max , where r max can be set to 150 km and r min = r/2 ( r is the radar range resolution) (Fig. 8).
The square root is introduced in order to ensure that quality does not drop too fast with the range distance.In the future, a more empirical approach will be adopted, evaluating the range distance quality for each radar on basis of a comparison with rain gauge measurements.

Vertical variability
As a result of the storm vertical variability, the radar observations made at relatively high altitudes are not representative when estimating precipitation at ground level.In order to deal with such an issue the reflectivity field can be projected onto the surface by estimating the so-called vertical profile of reflectivity (VPR), and this is the approach adopted in the present work.Thus, after correction for VPR, the associated quality is assumed equal to 1.

Attenuation
Rain path attenuation is one of the main impairments when estimating rainfall using frequencies higher than S band.While for dual-polarization systems there are a variety of possible solutions all based on the use of differential phase shift (Vulpiani et al., 2008), for conventional single-polarized radar the solutions are potentially unstable.For this reason it is recommended to evaluate them only qualitatively.The quality index associated with rain path attenuation can be defined as where PIA min = 1 dB and PIA max = 5 dB; PIA is the pathintegrated attenuation that can be computed from radar reflectivity Z (expressed in mm 6 m −3 ) as follows:  (11) Before evaluating the attenuation quality, it is recommended to remove unrealistic "spikes" in the field by applying a twodimensional median filter.Figure 9 shows the path-integrated attenuation for the sample radar image.

Overall quality
The final radar data quality can be retrieved by combining all the considered quality indicators.A multiplicative combination rule is proposed: where q k are defined in Eqs. ( 2) and ( 4)-( 8).The quality associated with the rain rate products at time t (Q R,t ) is the same as for the radar data (errors associated with the inversion process are not considered), while the quality associated with the cumulated rainfall can be estimated as the time average of Q R,t ; that is, with N t being the number of integrated rain rate fields.Figure 10 shows the overall quality for the sample radar image.
The quality index calculated with the methodology illustrated above is still not introduced in the common validation activity of the group.The present paper describes how it was used as a filter in the radar validation of the PR-OBS-3 product for 12 case studies, using radar data from the DPC radar "Il Monte": the radar pixels considered in the upscaling phase are all the ones for which the associated overall quality index exceeds the considered threshold.Those with quality index lower than the threshold are discarded.

Surface rainfall intensity calculation
In order to use the radar measurement for rainfall estimation, the surface rainfall intensity (SRI) is to be calculated, starting from raw data and applying a Z-R (reflectivity-rainfallintensity) relationship.It is worth remembering that any possible Z-R relationship choice is strongly sensitive to the drop size distribution variability.However, in the present work the Z-R relationship proposed by Marshall and Palmer (1948) has been adopted, because it is the most widespread in the operational radar community in Europe, even though it is more suitable for the estimation of stratiform precipitation.For the specific geographic area considered in the study, Vulpiani et al. (2012) found that the use of radar reflectivity for estimating precipitation is frequently subject to underestimation, mainly due to the orographic obstruction of radar beam, precipitation overshooting and attenuation.This is more pronounced when the lowest beam map (LBM) is the reflectivity product used for rainfall estimation.In order to reduce this effect, it was chosen to derive the SRI from the vertical maximum intensity (VMI), ground-projected by means of the retrieved VPR.
The quality index associated with the VMI map is, for each pixel, the quality index related to the PPI from which the pixel is extracted, calculated as described in Sect.4.6, Eq. ( 12).In the present work, the uncertainty related to the applied inversion algorithm has not been taken into account.

Upscaling of radar measurement on the satellite native grid and statistical scores calculation
The common validation methodology established for precipitation products implies the upscaling of radar data at the native grid of the satellite.While in some cases (microwavebased products) the aggregation of radar data has been weighted over a two-dimensional antenna pattern function, in this case (PR-OBS-3), due to the satellite product higher resolution, the upscaling is performed through a simple arithmetic average.All the radar pixels falling into one satellite grid cell are averaged, generating an upscaled value (Fig. 11).
Upscaling of radar images might introduce a potential error source in validation.The overall validation error E V would be then composed as where E R is the radar rainfall retrieval error and E U the upscaling error, appearing where satellite grid cells contain radar pixels for which no rainfall data are retrieved.Both errors are very difficult to estimate because of their strong local variability and dependency on rainfall type, season, orography, radar elevation and other factors.Yet there is a way to reduce both errors: E R can be reduced by introducing a quality index and filtering with it the radar pixels (see Sect. 3), retaining the ones with quality value above a selected validation threshold.E U , on the other hand, can be nullified by considering only the satellite pixels which are completely covered by valid radar pixels; that is, where all radar pixels have quality index above the selected threshold (this approach, however, strongly reduces the sample size).
During the routine validation activity of the H-SAF project, statistical scores are calculated on a monthly basis on the satellite-radar data pairs.Three different statistics are calculated: one for "land", one for "sea" and one for "coast" pixels.
The satellite measurements below 0.25 mm h −1 are considered as no rain.For the measurements above this threshold, three precipitation classes are introduced: low rain rate (< 1 mm h −1 ), moderate rain rate (between 1 and 10 mm h −1 ) and high rain rate (> 10 mm h −1 ).
The following statistical scores are calculated: -Continuous statistics (for 3 precipitation classes): mean error, standard deviation, mean absolute error, multiplicative bias, correlation coefficient, root mean square error and PR-RMSE (Puca et al., 2013).
-Multi-category statistics: contingency table (for 3 precipitation classes in the instantaneous products and 5 classes in the cumulated ones), probability of detection, false alarm ratio, critical success index (Puca et al., 2013); -Probability distribution function: 30-class distribution of satellite and radar data.
Every institute calculates the statistics over their surveillance area.An overall statistics for the entire H-SAF area is calculated by the DPC, as the coordinating institute, using the upscaled ground data provided by the member institutes.
The reference score for verifying the fulfilment of product requirements in H-SAF is PR-RMSE, defined as follows: Table 2 reports the user requirements to be achieved as a function of precipitation regimes in terms of PR − RMSE.The statistical scores typically calculated in the H-SAF precipitation product validation were evaluated for the different values of the quality index threshold.Some noticeable results were obtained, in particular regarding the fractional standard error (FSE), and the PR-RMSE, which is the reference score to assess the achievement of the product requirements defined inside the project.
As it can be observed in Figs. 14 and 15, the overall FSE and PR-RMSE calculated for the merging of all cases considerably decrease as the quality threshold increases.The same behaviour can be observed for almost all the single  cases (Figs. 16 and 17).Only two cases show practically constant results.This can be interpreted as the signature of a strong impact of the radar data quality on the validation of satellite-based rainfall estimation using radar data as reference.
These results indicate that the use of low-quality radar data in validating a precipitation product could lead to the product being underrated, wrongly stating that it does not reach the product requirements (which may result as correctly reached if the validation is performed with high-quality radar data).

Conclusions and future plans
A common procedure for validation of the blended IR-MW precipitation product (PR-OBS-3), as well as for other precipitation products in H-SAF, has been established and is already operational.Satellite-based rainfall estimations are regularly validated by a community of meteorologists and hydrologists belonging to eight countries in Europe, with different climatological, orographic and instrumental conditions for reference ground measurements.
In this framework, the importance of the definition of a quality index for non-polarimetric radar-derived rainfall intensities was clear.
Following a theoretical treatment of radar measurement uncertainty, here presented, a quality index has been developed and applied to a test radar site in central Italy.The present investigation was performed in order to evaluate how the introduction of a threshold in the quality index of the radar data used as reference impacts on the validation results of H-SAF satellite-derived precipitation products.The results here presented indicate a noticeable impact, with validation results showing better figures at higher radar quality thresholds.
In order to obtain more robust results about the impact on satellite-based rainfall products validation of the quality index introduction, and to extend the concept of radar-derived rainfall data quality index to polarimetric radars, the following steps are in progress: -Extension of the present investigation to a larger sample comprising rainfall events of different geographical areas.
-Evaluation of the impact of the quality index being introduced in validation of other satellite products (e.g.microwave-based products).
-Definition of a quality index also for polarimetric radar measurements, and evaluation of its impact on satellite product validation.

Fig. 2 .
Fig. 2. Geographical position and coverage of DPC radar "Il Monte", and elevation map of the surrounding region.The elevation in the color scale is expressed in meters a.s.l.

Table 2 .
User requirements for the PR-OBS-3 product.

Figure 16 :Fig. 16 .
Figure 16: Comparison between PR-RMSE depending on the quality threshold for the 12 798 case studies, over Land (top), Coast (centre), Sea (bottom) areas.Each line refers to a single 799 case study.800

Figure 17 :Fig. 17 .
Figure 17: Comparison between Fractional Standard Error depending on the quality threshold 830 for the 12 case studies, over Land (top), Coast (centre), Sea (bottom) areas.Each line refers 831 to a single case study.832 Fig. 17.Comparison between fractional standard error depending on the quality threshold for the 12 case studies over land (top), coast (centre) and sea (bottom) areas.Each line refers to a single case study.