The Hyogo Framework for Action, established 10 years ago, set out an ambitious framework for addressing disaster risk. While the platform was successful in reducing disaster mortality globally, there has not been similar success in tackling the underlying factors driving increasing exposure of people and assets to hazards (UNISDR, 2013, 2015). This is a goal of the subsequent Sendai Framework (2015–2030) and arguably requires a collaboration and integration between the disaster risk management (DRM) field and wider investment and development planning, in particular within the international development sector (Schipper and Pelling, 2006). At the same time, the cost effectiveness of ex ante risk reduction over ex post response is increasingly recognized (Mechler, 2016) and is increasingly relevant under conditions of funding scarcity (Frankenberger et al., 2014). Disaster resilience has come to the fore as an integrating concept by assisting in identifying novel ex ante strategies for integrated approaches to disaster risk reduction and response, as well as sustainable development. Disaster resilience definitions, frameworks, and approaches are being developed and promoted prolifically (Frankenberger et al., 2014; Winderl, 2014; Mitchell, 2013).

While resilience theories have informed wide-ranging disciplines for quite some time, an effort to identify operational indicators has gained some traction only in the last decade (Carpenter et al., 2005). Given the increased attention on enhancing disaster resilience, there has been growing investment in its measurement and the overall accountability of “resilience strengthening” initiatives. As identified by National Academies of Sciences (NRC, 2012) and Levine (2014) among many others, there is an urgent need for the continued development of theoretically anchored and practically applicable disaster resilience measurement frameworks and tools so that the field may (a) deepen understanding of the key components of “disaster resilience”, (b) enhance our ability to benchmark and measure disaster resilience over time, and (c) compare how resilience changes as a result of different capacities, actions, interventions, and hazard events.

The primary purpose of this paper is to present the Zurich Alliance community flood resilience measurement framework and associated tool, developed by an alliance of NGOs, academic institutions, and the private sector

The alliance members who designed and are managing the implementation of the tool are the International Federation of Red Cross and Red Crescent Societies (IFRC), the International Institute for Applied Systems Analysis (IIASA), the Wharton Business School's Risk Management and Decision Processes Center (Wharton), the international development non-governmental organization Practical Action, and Zurich Insurance Group who are also funding the endeavour.

Before delving into resilience measurement, we must first identify what it is we are attempting to measure; for disaster resilience this is not straightforward. “Resilience” has a long history, being adopted in different disciplines including engineering, psychology, ecological systems theory, and economics. In the disasters field, the central interest of this paper, resilience has a long and multidisciplinary history (for a review see Zhou et al., 2010). As the concept took hold in the disaster literature and practice, efforts to define it became a priority. Currently there exists a plethora of definitions of “disaster resilience” (see for example Adger et al., 2005; NRC, 2012; ABD, 2013; DFID, 2011; IPCC, 2012; Twigg, 2009). The field is characterized by lack of conceptual clarity and considerable debate (Zhou et al., 2010).

A central critique of resilience thinking is that it is a normative approach that accepts the system(s) as a given and works within it, crowding out space for questioning the underlying problems. Berkes and Ross (2013) identify lack of attention to power and agency as key critiques of resilience in DRM. That is, by focusing on existing community capacities, resilience thinking might miss important institutional arrangements that are limiting community capacity. Further to this is the critique that resilience is attractive to the “small government” discourse and is being used to justify shifting risk from government onto citizens (Welsh, 2013). Our approach to resilience, outlined below, starts with the current system before connecting with development and vulnerability theory to put people at the centre of decisions regarding their risk and wellbeing.

One of the early tasks of the alliance was to identify the research gaps in community flood resilience. As part of this research we conducted a review of resilience definitions, which is summarized in Keating et al. (2014, 2017). We then came to our definition of disaster resilience which underlies the measurement framework: the ability of a system, community, or society to pursue its social, ecological, and economic development and growth objectives, while managing its disaster risk over time in a mutually reinforcing way. Central to this conceptualization are the key community capitals – social, human, physical, financial, and natural – which holistically make up the socio-economic system (DFID, 1999; Keating et al., 2014, 2017). This conceptualization is centred on enhancing wellbeing as the goal of resilience, rather than disaster risk management, which can be a means to resilience. This puts focus on the interplay between disaster risk management and development trajectories, such that if one undermines the other then disaster resilience is not achieved. A key example of undesirable interplay is that of erosive coping – where actions taken to respond and recover from floods erode long-term wellbeing, potentially even trapping people in a poverty cycle (Heltberg et al., 2012). Alternatively, actions which are designed to enhance wellbeing, but lead to an unbalanced increase in disaster risk, are also undesirable.

Reviews and analyses of the state-of-the-art of disaster resilience measurement (Schipper and Langston, 2015; Winderl, 2014; Levine, 2014; Mitchell, 2013; Constas and Barrett, 2013) catalogue a plethora of offerings focused from the household to the national scale, from single hazards to multiple hazards to general resilience, and designed for different purposes. We, along with many others (Schipper and Langston, 2015; Levine, 2014), emphasize that there is no one-size-fits-all “resilience” measure, nor should there be. The development of various and varied resilience measurement frameworks should be seen as a positive step towards understanding resilience and operationalizing the concept. At the same time, many scholars and practitioners would like to know if there are any widely applicable capacities which provide for disaster resilience or if they are solely contextually and temporally specific.

Empirical evidence for the existence of such capacities is a critical gap in the field. Despite the proliferation of frameworks, Winderl's (2014) review concurs with many others that “no general measurement framework for disaster resilience has been empirically verified yet” (p. 19). This highlights a key challenge for any resilience-building efforts: if resilience cannot be empirically verified, how can we know we are measuring actual resilience? When most discuss “measuring resilience” they are really aiming to measure resilient capacity before an event. Stakeholders need to know before a risk event occurs whether they have effectively balanced risk and opportunity in building capacity to withstand and recover from the event. Related to this is the fact that civil groups, authorities, and NGOs would like to know whether activities they have implemented with the purpose of building resilience have achieved their goal. Winderl (2014) outlines the problem of circular reasoning when it comes to measuring ex ante: if we determine a priori which characteristics make a system resilient, then design interventions to enhance these, then measure again – we will necessarily find that resilience has been increased. However, with no empirical evidence to justify the selection of those characteristics in the first instance, we have no real evidence that resilience has actually been increased. As a latent property disaster resilience is only visible, or “revealed”, after a disaster has occurred (Schipper and Langston, 2015; Frankenberger et al., 2014), yet measuring impacts or outcomes in isolation tells us very little of what contributed to actual resilience. This is a critical distinction which is not always fully acknowledged in disaster resilience definitions and measurement frameworks.

However, this is precisely the distinction that is critical for empirically testing resilience measures. That is, to empirically test and validate a measure of resilience, it is required that (a) implementation of the measurement occurs across different contexts and is monitored consistently over time and (b) understanding and capturing both pre-event characteristics (what we call sources of resilience) and a set of pre-determined post-shock outcome measures occur, in order to learn which sources are most effective for the realization of resilient outcomes. We know of no disaster resilience measurement frameworks or programs which do this. It is this gap that the Zurich Alliance community flood resilience measurement framework and associated tool presented below aims to fill in the context of community flood resilience. The content of our measurement framework has been drawn from the existing literature (providing theory and evidence) and alliance members and peer group expertise. It is an indicator-based approach consisting of 88 measures of potential sources of resilience (listed in Appendix B) and 29 post-flood outcome measures (also listed in Appendix B). At the time of writing the tool is being tested by six NGOs, in approximately 75 communities across eight country programs. The initial programme will allow for preliminary empirical testing and feedback from implementers.

The paper proceeds as follows: first we review current thinking on measuring resilience and draw out pertinent challenges, particularly relating to empirical validation. We then describe the measurement framework development process. Next we present the content of the framework and then describe the operationalized tool. In the conclusion we present our path forward for undertaking much needed empirical analysis to understand community flood resilience. By documenting our process, we hope that others looking to measure resilience, or develop a resilience measurement framework of their own, can learn from our experience.

Several national and international aid agencies have proposed versions of resilience indicators (Alinovi et al., 2009; USAID, 2013) and a number of regional disaster resilience indicators have also been developed (Cutter et al., 2010; Resilience Capacity Index, 2017). Twigg's (2009) Characteristics of Disaster Resilience Community is designed for, and in cooperation with, NGO and civil society organizations; it systematically and extensively explores many factors which may contribute to disaster resilience. More recently the BRACED project has proposed a composite index to measure resilience based around tracking adaptation, anticipation, absorption, and transformation (Bahadur et al., 2015).

The last few years have seen a number of reviews of disaster resilience measurement frameworks (Schipper and Langston, 2015; Winderl, 2014; Ostadtaghizadeh et al, 2015; Oddsdottir et al., 2013). Schipper and Langston (2015) review 17 sets of indicators from the household to the national level. We do not aim to reproduce such a review here, but instead draw on these and other critical analyses to highlight some of the challenges associated with measuring disaster resilience. There are many substantial differences between national versus local-level disaster resilience and associated measurement frameworks. Because the framework presented here is at the community level, our review focuses on issues and challenges at this scale. There are few examples of comprehensive community disaster resilience measurement frameworks available, only a handful of which have been implemented in the field and none that are empirically validated (Winderl, 2014); this is because measuring resilience is not straightforward for two main reasons: (1) it is a latent quality that is not revealed until tested and (2) the characteristics that influence this latent quality are often a complex set of holistic and qualitative characteristics.

Measuring latent and qualitative characteristics is not a challenge unique to resilience measurement. Gathering and synthesizing subjective (opinion) and/or qualitative (narrative) information is a mainstay of modern development monitoring and evaluation (M & E) practice. The Resilience Measurement Technical Working Group has undertaken a comprehensive review of, and developed a position on, qualitative data and subjective indicators for resilience measurement (Constas et al., 2015). They find that qualitative and subjective measures are essential to resilience analysis, particularly in relation to social factors. They suggest mixed methods approaches and that solid planning (including resourcing) are required to undertake effective qualitative data collection. Below we describe how our framework and tool tackles this challenge.

Thus community disaster resilience measurement frameworks or indicators – including ours – share many common theoretical and practical challenges. First, in doing this work, initial questions immediately confronted are (1) defining an appropriate scale of analysis both geographically and temporally – specifying boundaries such as “resilience of what to what?” (Carpenter et al., 2001) – and (2) identifying the potential end users (“indicators for whom?”) and potential purposes (“indicators for what?”). This helps bring clarity into the complex process of measurement framework development (de Sherbinin et al., 2013). Without these specifics it becomes increasingly difficult to distinguish disaster resilience measurement frameworks from general development assessments. A key challenge, however, is balancing the need for specific indicators (to a particular hazard in a particular place for a particular institution) and the need for wide applicability. A related further challenge is that resilience to one hazard does not necessarily translate into resilience to another. In fact, it is entirely possible that measuring and enhancing resilience to one hazard may inadvertently reduce resilience to another (Schipper and Langston, 2015).

Particularly pertinent to our community-level framework is the observation by several authors (Béné et al., 2012; Frankenberger et al., 2014) that resilience at the community level is dependent upon changes at lower “levels”, i.e. individuals and groups within communities, as well as changes at higher municipality, national, or even global levels, which are outside the scope of direct NGO community programming initiatives. NGOs are often limited in their capacity to activate change at these scales. “Community” itself is a particularly elusive concept without clear definition. It is a highly contested term and has been critiqued from multiple perspectives. Particularly relevant to disaster resilience is the inherent tension between discourses of empowerment, which dominate the rhetoric, which have “concealed a much more conformist and conservative reality” (Shaw, 2007, p. 26) borne of the paternalistic history of the development field. Our definition of community is outlined in Sect. 3.

As communities are dynamic complex systems, Levine (2014) argues that the modular approach to measuring resilience, such as measuring a set of discrete characteristics, “assumes that improvements in any component of resilience score are of equal importance” (p. 8). As he suggests, this is deeply problematic when thresholds are present: an increased score in some areas might not actually afford a benefit until it reaches a certain threshold or unless it is combined with some other characteristic or capacity. A further challenge is that resilience can change over time depending on changes both within and outside the system (Holling, 2001). The extremely scale-, place-, and system-specific nature of capacity profiles also creates difficulties when attempting to generalize a set of key factors which enhance resilience (Tol and Yohe, 2007; Vincent, 2007).

Lastly, Béné et al. (2012) and Levine (2014) put forward the critique that a resilience-focused perspective runs the risk of diverting attention away from the most vulnerable or marginalized groups in a community, in favour of a more average community-wide perspective. We must be cognizant of the fact that building the resilience of the majority does not necessarily meet the development needs of the poorest. Resilience is not a pro-poor concept, in that it does not exclusively apply to, or benefit, the poor. As such, resilience building does not replace poverty reduction (Béné et al., 2012) but can rather be considered as one element in protecting current and future development gains.

The Zurich Alliance community flood resilience measurement framework has been pilot tested, reviewed, and revised, and now a beta version has been adapted into a web-based platform and integrated mobile data collection application, via which implementers collect data and assign grades for each of the indicators (pre-event “sources of resilience” and post-event impact assessment).

The conceptualization of disaster resilience underlying the framework is set out in the introduction. Narrowing this down, for the purposes of this framework we are specifically interested in the resilience of communities to flooding. While acknowledging that national and global drivers play a significant part in flood resilience, the community level is where flood impacts are felt most viscerally, where much action on flood resilience needs to be taken and are the focus of many flood and development activities. The NGO practitioners on the design team put forward the definition of community used here. A “community” could be defined geographically (perhaps in rural contexts) or by administrative boundaries (which may work in more urban situations). However, no single community will “feel” like another and there are cultural aspects to consider, too. As a result we have concluded that when it comes to ground reality a community largely defines itself.

This measurement framework applies specifically to community-level resilience to flooding and is designed to (a) help guide NGO community development and flood-based programs and (b) to provide a platform for empirical analysis of resilience. As Frankenberger and Nelson (2013) point out, an approach to resilience measurement which measures ex ante but does not test whether what was measured was relevant once the event occurred cannot make any claims about having measured resilience. By tracking the sources of resilience both pre- and post-event, together with outcomes in the event of a flood, we can observe how development, disasters, and DRM activities occurring in the community are eroding or supporting wellbeing. Over time and studies in different contexts, this testing can build an evidence base for what actually makes the difference for community-level resilience to flooding.

The 88 sources of resilience have been developed with the underlying frameworks of the five capitals (5Cs) and the 4Rs. These were then cross checked by categorizing them within three other operational frameworks commonly used by NGOs. These provide a cross check not only that we are measuring a holistic set of sources but also that the suite of categorizations attached to each source provide for analytical depth by allowing for multiple perspectives on the results. These also increase the ability to communicate results to many different stakeholder audiences in the contexts within which they are working. We provide examples of each of these categorizations below.

Before exploring dimensions of the tool in detail, we provide Fig. 3 as a reference point. The figure shows the steps that are undertaken when using the framework and where different aspects fit. The first step is data collection (see Sect. 4.3), which can be done using various methods. After data have been collected, experts assign grades to each of the 88 sources of resilience, according to the source rationale and the A–D grade definitions. Each source has already been assigned (tagged) to the five categorizations (Sects. 4.1.1 to 4.1.5). Once all 88 sources are graded, results are generated according to the different categorizations.

To test the framework at scale and collect the data required for undertaking empirical analysis of community flood resilience, this framework was implemented into a community flood resilience measurement tool – an integrated, hybrid web-based and mobile device system for creating questionnaires based on the flexible combination of data collection methods for each source, assigning data collection work, collecting data, undertaking grading, generating outputs, and storing data on a (protected) central database. The six NGOs implementing the tool have been trained in understanding the framework and using the tool. Over 2 years they will collect baseline, end line, and outcome measures (if a flood occurs) in 75 communities in eight countries. A train-the-trainer workshop was conducted with approximately 20 NGO staff from 9 to 13 November 2015. During this week the staff were trained to (a) implement the tool and (b) train their colleagues to implement the tool.

Implementing the tool includes being confident in ones understanding of the rationale behind the tool, being able to consistently grade sources, interpret outputs, and use the internet and mobile applications. It also critically requires a sound understanding of how the tool outputs should, and should not, be utilized. The measurement tool does not replace existing processes, but it does foster an open dialogue around findings. It is designed to help identify potential areas for intervention, but choosing an intervention is a much more complex process which must consider multiple factors and perspectives.

As with all information, the implementing NGO is obliged to feed information back to the community in an accessible and responsible manner. While the final decision rests with the NGO, we strongly advise against feeding back information on quantitative scoring because of the specialist training required to accurately interpret grades. Grades on their own are largely meaningless; what is important for analysis and communities is how different sources interact and how resilience is changing over time in that community. Pilot testing to date has shown that the information gleaned from the measurement process, presented in a qualitative way, can provide a very useful starting point for discussions with communities about flood resilience.

Internal communication must also be carefully considered. There is the risk that the generation of a grade may result in competition between project managers or field workers. It is up to the core assessment team to contextualize the measurement process and resulting grades within the wider context of their community development work to ensure that the grades are understood. There may be a temptation to gravitate towards interventions which one believes will increase the resilience grade with the least amount of resources or effort. While one of the benefits of the tool is that it might identify relatively cost-effective “quick wins”, it should never be the case that the objective of the intervention becomes the increase of the resilience grade per se.

For the purpose of the empirical research, all data which go into the tool system are effectively anonymized. All respondents provide informed consent before information is collected, and no individual can be identified from the data (either in its raw form or aggregated).

Schipper and Langston's (2015) review and expert consultation report argue that measurement frameworks are “based on assumptions about how systems work, albeit informed assumptions” (p. 19). Measurement frameworks make assumptions about the way in which the presence or absence of each indicator (often measured ex ante) will impact resilience ex post. If this project is to achieve its aim of empirically exploring the critical sources of resilience, we need to do more than just measure sources in a number of communities; we need to test those measures against outcomes. This is why we are collecting post-flood outcome data – so that the original sources can be compared to outcomes in order to understand what really made a difference in the flood event and recovery (i.e. actual resilience).

The implementation of the measurement tool described above will generate an unprecedented database of community-level information and experience with flooding, collected in a consistent way. Data will be augmented by information from implementing NGOs regarding the community context, major events, or changes in the community during the study period, and interventions implemented. Analysis of these data will lead to insights about community flood resilience generally: what typical community profiles, patterns in strengths and weaknesses across the different categorizations, correlations between different capacities, and effectiveness of different intervention types are.

Our analysis of the validity of the sources of resilience within the framework – our hypothesis from research to-date – will take a multi-pronged approach. Firstly, we will explore the empirical relationship between the sources (recorded in the baseline) and outcomes (recorded in the post-flood outcome measurement). A suite of quantitative data analysis methods will be used to undertake this endeavour. Furthermore, each time a source is graded (75 communities at baseline and end line), assessors are asked two question about the source:

Were you confident grading this source? (Y/N)

Preliminary implementer feedback at the time of writing has indicated that the process of training staff to use the tool, and implementing the tool, is already producing positive outcomes. In particular, implementing NGOs have reported that the holistic view of the community system is building local staff capacity to think systemically about their work. For example, in addition to the traditional physical infrastructure, the human and social elements required to make an early warning system operational are being considered early. The data collection effort holds many co-benefits for supporting other NGO work beyond a flood focus. The data collection technology is considered superior in its accuracy and efficiency to traditional paper-based approaches.

NGOs implementing the measurement tool have expressed a desire for a more explicit handling of power, indicating a preference for a sixth “political capital” to be added to the five capitals framework. Similarly they have indicated the need for more explicit incorporation of social inclusion/exclusion in the measurement framework. These are all recommendations which should be considered carefully by the field as it move forward.

Via this process we hope to be able to generate evidence for which of the sources of resilience, if any, make a difference across diverse contexts. It is possible that community flood resilience is entirely contextually specific, or it boils down to a few core sources of resilience meeting a certain threshold; this is what we will explore. The extensive testing phase is also providing a platform for the formation of a peer group of practitioners working on community flood resilience and using the measurement tool. The insights this peer group generates together provide a critical feedback to resilience measurement and practice, as well as builds practitioner capacity. In the future reflection on this extensive process will allow critical reflection on many of the challenges associated with measuring resilience.

Winderl (2014) concludes his comprehensive review of measurement frameworks with a set of recommendations or lessons, presumably for the development of future frameworks or iterations of existing ones. These recommendations are theoretically sound but would be impractical to implement within one framework. When working in such an applied field, and relying on collaboration from communities, NGOs and other non-research institutions, there is little space for idealism in research design. Our endeavour to date has highlighted that capacity and willingness to implement complex theoretical frameworks are very limited at the grassroots NGO level. There is a need to appreciate that staff resources and budgets are very limited, and implementing idealistic frameworks in a time-consuming manner will distract from any interactions and bringing tangible benefits to the communities. We are also keenly aware that communities are not there as research test beds to study whether resilience can be observed; rather, testing frameworks need to bring a benefit to the communities working with NGOs and researchers.

The measurement framework presented here is by no means perfect; the design team sacrificed a number of “nice-to-haves” in order to design a framework and tool which is both functional and analytically useful. We found that designing the framework and tool in an alliance between practitioners, academics, and risk engineers has contributed to optimizing it within practical constraints. Our long-term vision is to develop a replicable and scalable approach to measuring community flood resilience. The approach described here could also be adjusted to apply to different perils and different levels of social organization. By utilizing a consistent framework, information on resilience is comparable and insights can more readily be identified.