Agricultural drought, which occurs due to a significant reduction in the moisture required for vegetation growth, is the most complex amongst all drought categories. The onset of agriculture drought is slow and can occur over vast areas with varying spatial effects, differing in areas with a particular vegetation land cover or specific agro-ecological sub-regions. These spatial variations imply that monitoring and forecasting agricultural drought require complex models that consider the spatial variations in a given region of interest. Hierarchical Bayesian models are suited for modelling such complex systems. Using partially pooled data with sub-groups that characterise spatial differences, these models can capture the sub-group variation while allowing flexibility and information sharing between these sub-groups. This paper's objective is to improve the accuracy and precision of agricultural drought forecasting in spatially diverse regions with a hierarchical Bayesian model. Results showed that the hierarchical Bayesian model was better at capturing the variability for the different agro-ecological zones and vegetation land covers compared to a regular Bayesian auto-regression distributed lags model. The forecasted vegetation condition and associated drought probabilities were more accurate and precise with the hierarchical Bayesian model at 4- to 10-week lead times. Forecasts from the hierarchical model exhibited higher hit rates with a low probability of false alarms for drought events in semi-arid and arid zones. The hierarchical Bayesian model also showed good transferable forecast skills over counties not included in the training data.

Drought is a naturally occurring phenomenon that affects the food security of approximately 55 million people annually and can severely impact a country's economy

Agricultural drought, which is the focus of this paper, is known to be very complex

Its onset can be slow and can occur in vast areas with varying spatial impact

Drought EWSs have been recognised by global initiatives like the United Nations Sustainable Development Goals (SDGs) for effective drought monitoring and hazard preparedness

Recent advances in computational power and processing hardware have enabled researchers to develop and deploy machine learning models

This paper is part 2 of a previous study

Figure illustrating the concept of no pooling, complete pooling, and partial pooling of the data.

Although known to vary over the different regions, the effects of biophysical indicators on vegetation also show some similarities across the different regions

The HBM is an extension of the regular Bayesian regression, where model parameters differ based on the variations within a given dataset

Another advantage of using the HBM is its transferability

Our objectives for this proof of concept are to

improve the forecast accuracy and precision of the Bayesian auto-regression distributed lags (BARDL) model with a hierarchical Bayesian model in regions with diverse AEZs and land covers;

test the transfer learning property of hierarchical models that enables pre-trained models to be used on similar data from a different location without the need to retrain the model

To test our concept of forecasting vegetation condition with HBM, we sampled data from some selected counties in Kenya (Baringo, Kitui, Marsabit, Narok, Tana River, Turkana), shown in Fig.

Maps of Kenya showing agro-ecological zones (AEZs) and land cover maps for the counties from which pixels were sampled. Kenya AEZ boundary map credit: IGAD Climate Prediction and Application Centre (ICPAC). Land cover map credit: European Space Agency (ESA), Climate Change Initiative (CCI).

We selected only six counties because the algorithm used for parameter sampling by the HBM can be very time-consuming when the input data are more than 10 000 records. The sampling time is also mainly due to the complex nature of HBMs.

The precipitation data between 2001 and 2018 were obtained from the Climate Hazards Group InfraRed Precipitation (CHIRPS) project

The daily soil moisture products by the European Space Agency's Climate Change Initiative (ESA-CCI), from 2001 to 2018, were used for this work. The data represent soil moisture at a 10 cm soil depth and are derived from an algorithm that takes information from multiple active and passive synthetic aperture radar (SAR) satellites

The vegetation index (VCI) used as a drought indicator for the work was derived from the bidirectional reflectance distribution function (BRDF)-corrected MODIS product, MCD43A4 Version 6

Two HBMs were developed in this study, one based on AEZs and the other on land covers. AEZs are geographical areas characterised by similar climatic patterns and soil moisture levels suitable for agriculture and vegetation growth. These zones were created by the Food and Agriculture Organization (FAO) in collaboration with the International Institute for Applied Systems Analysis (IIASA) and are based on a framework that utilises a series of models with climate and land use information to quantify and map out the regions

The AEZs in our study area can be seen in Table

Table describing the agro-ecological zone, vegetation type, and annual rainfall levels.

Source:

Most drought-prone regions are made of diverse vegetation covers; these include tree covers (forests), grasslands, shrubs, and croplands. The impact of drought on these land cover types varies both spatially and temporally. Thus, a drought forecast model should consider the varying effects of the biophysical factors on the various land covers. Using an HBM framework in this context enables us to achieve this. Data corresponding to the various vegetation land covers were extracted with the 2016 Sentinel 2 land use and land cover (LULC) map

Visit

A major challenge with using optical EO images is cloud cover and cloud shadows. In addition, pixel reflectance values sometimes fall outside the meaningful range due to errors during the atmospheric and radiometric correction process. These clouded and poor-quality pixels were filtered out with the quality assurance maps that come with the images. Weekly averages of VCI, precipitation, and soil moisture data corresponding to the vegetation land covers of interest were extracted from the selected counties using the European Space Agency (ESA) 2016 Sentinel 2 land use and land cover (LULC) map. The same data within the various AEZs were also extracted using the AEZ shapefiles produced by the IGAD Climate Prediction and Application Centre (ICPAC) (

The HBM implemented in this work was done via an auto-regressive distributed lag (ARDL) model

An illustration of the parameter structure of the hierarchical Bayesian model based on partially pooled data (

The Bayesian framework used for the parameter inference is based on Bayes' theorem in Eq. (

The HBM will enable us to fit the ARDL model by simultaneously inferring global parameters (Nodes A and B in Fig.

A directed acyclic graph (DAG) schema representing the hierarchical model based on varying agro-ecological zones. The figure depicts how the hierarchical model parameters and input data are defined and structured.

The HBM was based on an ARDL (

Parameter approximation for the HBM was sampled with the Hamiltonian Monte Carlo (HMC) algorithm

The hierarchical BARDL model in this study was defined as

Below (Fig.

The nodes seen in the HBM DAG in Fig.

Node A is the global (mean) regression intercept or (

Node B represents global (mean) regression coefficients for each of the lagged input variables (precipitation and soil moisture) or (

Node C represents the Cholesky covariance matrix used as hyper-priors for the group-level

Nodes D and E are the Cholesky standard deviation and correlation from the matrix decomposition, respectively.

Node F represents the offset distribution (Gaussian) hyper-prior to make the model non-centred.

Node G is the prior group-level parameters for

Node H represents the error term in the HBM regression.

Node I is the likelihood function (Eq.

Node J is our lagged inputs datasets.

Node K is the categorical values that map the input data to their respective AEZs.

Node L is the observed VCI3M values at an

The forecast method used in this work was the direct multi-step forecast approach as described by

With this approach, separate models are fitted for every

After the parameter estimation via HMC sampling, the held-out dataset is passed to the fitted model (without the target variable) to produce forecast values for

The forecast uncertainties were analysed with the mean prediction interval width (MPIW) and the prediction interval coverage probability (PICP)

The MPIW was derived as follows:

The PICP was derived as follows:

Other forecast verification metrics used in this paper are the receiver operating characteristic (ROC) curve

The ROC curve tells us the likelihood of a forecast being true (true positive rate – TPR) for the given drought threshold and the probability of the forecast event being false (false alarm rate – FAR). In addition, the area under the curve (AUC) was also computed to determine the propensity of our model to separate drought events for the set threshold

The reliability diagram allows us to assess the accuracy of the forecast probability predicted by our model. The probability of a drought event is computed using the full posterior distribution of our forecasts at a given drought threshold. The same threshold is used to convert observed data into binary events, where 0 indicates a “no drought”, and 1 indicates a “drought” event. The forecast probabilities and observed binaries are binned into probability intervals and used to plot the forecast reliability diagrams. The reliability of the forecast is assessed by the number of times an observed event agrees with a given forecast probability

Our dynamic HBM for forecasting VCI3M was tested on datasets based on their AEZs and vegetation land covers. Two models were developed, a BARDL model based on a no-pooling approach as a base model and an HBM based on the partial-pooling approach. The BARDL model was used to forecast VCI3M for the different AEZs, referred to as “BARDL-AEZ”, and different land covers, referred to as “BARDL-LC”. The HBM, which was modelled with partially pooled AEZ data, is referred to as “HBM-AEZ”, and the model-based partially pooled land cover data are referred to as “HBM-LC”. The results shown in this section are a comparison of BARDL-AEZ to HBM-AEZ and BARDL-LC to HBM-LC.

Plots showing

Plots showing

The AEZ-based models were used to forecast VCI3M for the humid, semi-humid, semi-arid, arid, and very arid zones. The

Plots showing

Figure

Forecasts by both the HBM-AEZ and the HBM-LC were also evaluated for long rain (March, April, May – MAM) and short rain (October, November, December – OND) seasons. In both seasons, the HBM-AEZ model gave higher forecast accuracies across all lead times as seen in Fig.

Plots showing forecast for arid zones for 4- and 10-week lead times and their uncertainties (PICP and MPIW).

The forecast uncertainty in both forecast models was analysed using the PICP and MPIW. The desired PICP value usually ranges between 0.90 and 0.99

The mean PICP and MPIW for both AEZs and land covers over the selected counties are in Tables

Although our models produce accurate VCI3M values at the various lead times, our target users are also interested in whether or not a drought event alarm will be triggered at a defined threshold. Therefore, the skill of the forecast models at predicting drought events was assessed with the ROC curve with a threshold of VCI3M

The ROC plots in Fig.

ROC plots generally showing higher hit rates for HBM in semi-arid, arid, and very arid zones.

Figure

ROC plots for crops, grass, and shrubland covers.

The reliability plots in Fig.

Reliability and sharpness plots showing a joint distribution of forecast probabilities and observed frequencies for various arid and very arid agro-ecological zones for the different lead times.

The reliability diagrams for both BARDL-AEZ and HBM-AEZ (Fig.

The skill of the models at predicting the onset and end of a drought period can be seen in Fig.

A time series plot showing the observed and forecasted VCI3M for the period of 2017. Forecast probabilities are indicated as points on the horizontal lines marking the onset and end of a drought period.

Although the data used for training and developing forecast models are usually sampled to represent a given area of interest, the goal in most cases is to have models that can scale up to produce forecasts over more expansive areas. The second objective of this study was to test the transfer learning capability of HBMs over other regions. The partially pooled data used for hierarchical parameter approximations were sampled from six counties. The trained models for the different lead times were then used to forecast VCI3M for the AEZs and land covers over 10 additional counties (shown with black boundaries in Fig.

Plot showing

In this paper, we seek to improve the forecast accuracy of VCI3M over vast areas with varying AEZs and land covers using an HBM. Compared to the non-hierarchical BARDL model, the HBM presented a more realistic approach for forecasting VCI3M in regions with different AEZs or land covers. The evaluation of the HBM based on

The strong relationship between lagged soil moisture VCI3M over forest areas could be due to the frequent precipitation and high soil moisture retention in areas as seen in Fig.

Overall, results from the various skill assessments showed that forecasts with HBM were more precise, with a lower false alarm rate for drought events than the BARDL model. The HBM was also able to effectively identify drought events in counties with diverse AEZs and some land covers. The HBM also performed well in both the long and short rain seasons for the arid and very arid AEZs, which are more prone to drought occurrences.

Relating the overall forecast skill assessments from this work to previous works, the HBM showed an approximately 1-week increase in the forecast range compared to the results from the BARDL method used in

Aside from the improvement in the forecast range, the HBM also had some added strengths. First of all, the hierarchical nature of the model parameters (see Fig.

The threat of agricultural drought to food security and global economies has pushed agencies like the USAID and FAO to develop early warning systems that continually monitor drought events. However, agricultural drought over vast and diverse arid and semi-arid lands (ASAL) regions poses a challenge to effective monitoring

The methods used in this paper also had a few limitations. A fundamental limitation was the timely availability of the ESA CCI soil moisture data, a setback that can affect the prospects of producing real-time forecasts. Parameter inference via the HMC sampler also takes a long time to complete partly due to the complex nature of the HBM and the number of data points involved. However, this was not considered to be a significant limitation as it only occurs during the model training phase. Once the model converges, and sampling completes, the posterior predictive sampling or forecasting of VCI3M takes seconds.

In this paper, we present a proof of concept that HBM can factor spatial differences into drought forecasting. Using this approach also allowed us to understand the vegetation dynamics in agro-climatic areas and regions with diverse vegetation covers. For instance, we saw an approximately 1-week gain in forecast range for vegetation conditions in very arid areas as well as forests (tree cover) and cropping areas. Furthermore, we show that soil moisture contributes more when forecasting VCI3M over very arid areas and forest covers. However, future work on drought forecasting should explore other indicators like the vegetation health indicator (VHI) or VCI based on the soil adjusted vegetation index (SAVI) instead of NDVI as demonstrated by

We also show that HBM trained with data in one area could be transferred to other similar datasets in other regions. Future research work should consider more complex HBMs that take into account variations for different land cover types within the various agro-ecological zones and the seasonal differences.

Plots showing

Plots showing

Table showing a PICP and MPIW (in parentheses) for the various agro-ecological zones.

Table showing a PICP and MPIW (in parentheses) for the various vegetation land covers.

Reliability and sharpness plots showing a joint distribution of forecast probabilities and observed frequencies for various agro-ecological zones and land cover for different lead times.

Plots showing the relative importance of the lagged input variables (VCI3M, P3M, SM3M) and VCI3M at 4- to 12-week lead times for the different agro-ecological zones.

Plots showing the relative importance of the lagged input variables (VCI3M, P3M, SM3M) and VCI3M at 4- to 12-week lead times for the different vegetation land covers.

The data and code repository can be found here:

EES is the lead author and was responsible data pre-processing, modelling (coding), and running BARDL and HBM. JMM assisted with data acquisition, pre-processing, cartography, and feedback. AB developed the code for the smoothing algorithm used for the time series data. SO, PR, and PH conceptualised the initial idea and provided supervision and feedback. The final manuscript was edited and reviewed by all authors.

The contact author has declared that none of the authors has any competing interests.

Publisher's note: Copernicus Publications remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The work was funded by the UK Newton Fund's Development in Africa with Radio Astronomy (DARA) Big Data project delivered via STFC with grant number ST/R001898/1 and by the Science for Humanitarian Emergencies and Resilience (SHEAR) consortium project “Towards Forecast-based Preparedness Action” (ForPAc;

This research has been supported by the Newton Fund (grant no. ST/R001898/1) and the Natural Environment Research Council (grant no. NE/P000673/1).

This paper was edited by Maria-Carmen Llasat and reviewed by three anonymous referees.