Evaluation challenges for climate adaptation

Evaluation can generate critical evidence about the effectiveness of climate and health adaptation, as well as insights about how to improve program performance [9,18]. Evaluation is a systematic way of asking “Are we doing the right things? And are we doing the things right?” [26,27]. Aptly, there is increasing demand for “evidence-based public health” that develops, implements, and evaluates the effectiveness of programs and policies through scientific reasoning [11,28,29]. There is also growing consensus for explicit consideration of who receives those benefits [30–32]. Standard health monitoring and evaluation practices and indicators, however, are ill-equipped to track and ultimately be used to enhance system-level climate resilience [28]. Despite the urgent need, there is sparse evidence for the health benefits of climate adaptation [11,33]. However, with climate change impacts becoming more frequent and severe, there is increasing demand for evaluations of adaptations to quantify health benefits [34].

The challenges of conducting evaluation in a climate and health context are many and parallel those experienced by adaptation planners in general. Some of the most formidable include dynamic baseline conditions; the need to measure impacts over multiple overlapping time scales, across sectors, and in the context of multiple, interacting, up-and down-stream moderating and mediating factors; inherent uncertainty in the rate, magnitude, and effects of climate change for a given location; and the contingencies poised by current climate mitigation policy decisions [18,27,28,35–37]. Attribution is often difficult in evaluation but is especially so in climate adaptation due to the long term and multifaceted set of influences outside of a single program or policy. Moser and Ekstrom identified specific organizational barriers to evaluation of climate adaptation that included perceived need and feasibility of evaluation; availability of funding, expertise, data, and methods; and willingness to learn. In addition, they noted a constellation of barriers around the willingness, feasibility, and legal aspects of revisiting prior decisions [38].

Another perennial evaluation challenge is assessing maladaptation. Ultimately, adaptation success depends not only on how it meets its intended goals but, crucially, how it affects the ability of others, present day or in the future, to meet theirs [39,40]. Barnett and O’Neill have identified five manifestations of maladaptation: increase of greenhouse gasses, disproportionate burden to those most at risk, high opportunity costs, reduced inventiveness or capacity to adapt, and decisions that limit future choices and thus increase vulnerability [41]. Fundamentally, maladaptation may be more likely when critical drivers of climate change vulnerability are poorly understood or not considered when the project scope is too narrow [36].

Because evaluation in health-focused climate adaptation requires many considerations beyond those typically addressed in traditional public health programs, multiple frameworks have been developed to guide evaluators. Most have been developed in the context of international development programs and originally intended for middle- and low-income countries [24,27,36,42–51]. While some standard evaluation practices such as use of logic models seem to be commonly incorporated by health-focused adaptation evaluation frameworks, often additional tools and approaches responsive to the complexity of the field are included [24,42–45,47–49,51]. For example, several frameworks are cross-sectoral [24,42–45,47–49] and some contain guidance on adaptation practice [24,42,46,49,50]. To our knowledge these frameworks have not been widely used in the US context, suggesting a missed opportunity.

The climate ready states and cities initiative

In the US, CDC’s model for climate and health adaptation, BRACE, has primarily been implemented through the Climate-Ready States and Cities Initiative (CRSCI) [52,53]. This is the largest source of domestic climate and health funding from the federal government. The 2016–2021 funding period awarded annual grants ranging from $100,000 to $250,000 [54] to 16 state and 2 local health departments. All recipients were funded to implement BRACE in their local jurisdictions. CDC encouraged recipients to use the CDC Framework for Program Evaluation to guide their evaluation practice [55]. This framework aims for evaluations to be more contextualized and participatory to encourage use of the findings. The Framework involves six steps and five standards for effective, accurate, useful, feasible, and ethical evaluation. Steps place special emphasis on setting the appropriate evaluation focus, by engaging stakeholders and clearly describing the program to be evaluated. The last step prompts practitioners to ensure that lessons learned are shared and findings used. The framework conceptualizes evaluation as integral to a cycle of continuous program improvement [56].

Given that evaluation in the context of climate and health adaptation is challenging and there is not yet full consensus about the best ways to undertake this work, we can look to the CRSCI program as one example of practitioners tasked with a common purpose, through the CDC grant. The evaluation experiences of CRSCI recipients provides an opportunity to understand the real-world challenges and opportunities of frontline climate and health evaluation practice. The objectives of this paper are to describe the adaptation actions that were evaluated among CRSCI recipients and the approaches used, present a series of case studies on a subset of evaluations reflecting key challenges and opportunities, identify promising evaluation strategies, and propose how these insights might be applied to strengthening climate and health adaptation evaluation for future recipients and practitioners in the field. Our guiding questions were: 1) How have CRSCI recipients conducted evaluations of climate and health adaptation actions? 2) How have CRSCI recipient evaluation practices been similar or distinct? 3) What evaluation challenges and barriers have recipients encountered? 4) What factors facilitated evaluation?

Data collection

First, to compile the CRSCI evaluation dataset, we reviewed 54 Annual Performance Reports (APRs) and 72 sets of performance measures (PMs) that CRSCI recipients submitted to CDC as part of the annual grants management process. APRs and PMs are intended to ensure accountability in grant implementation and help CDC staff provide technical assistance to the recipient. They also provide an opportunity for an overall assessment of program accomplishments and impact. APRs follow a prescribed narrative format that includes work plans and a review of accomplishments. PMs include specific quantitative and qualitative metrics across four categories of recipient activities: capacity building; interventions; communications; and evaluation. For this project, we included APRs submitted from 2018 to 2020 and PMs submitted from 2018 to 2021, based on the availability of data at the time of the analysis. The grant cycle considered in this review concluded in August 2021. We reviewed the PM category of evaluation and the APRs to identify any evaluation-related activities. Evaluations did not have to be completed to be included.

The second dataset consists of select case studies of CRSCI evaluations. The principal aims of the case studies were exploratory and descriptive [59]. We selected multiple case studies to enhance external validity of the inquiry [60]. To select the cases, we applied three of Patton’s 16 purposeful sampling principles. We sought evaluation cases that were 1) reflective of evaluations conducted by CRSCI recipients (typical case), 2) completed and included a plan for use of the results (criterion), and 3) diverse in the climate hazards addressed (variation) [61]. The abstracted data on evaluation activities, described above, were used to guide the selection of case studies according to criteria suggested by the purposeful sampling principles. The case studies selected were: New York City’s Be a Buddy Program; Maricopa County Arizona’s Heat Toolkit Distribution; and Sarasota County, Florida’s Emergency Management Building Resilience Against Climate Effects Workshops.

To develop the case studies, we collected additional in-depth information from the project teams leading the evaluations through a standardized template, presented in the supporting materials (see S1 Text). The template consists of a series of prompts, to which teams initially responded in writing or via guided discussions. The prompts covered key elements of the intervention and the evaluation, such as design, methods, evaluation questions, findings, challenges, and facilitators. The template provided examples of facilitators (i.e., “staff, skills, relationships, champions, leadership, or strengths of the program”) and “challenges” (e.g., “staff, skills, access to data/information/population, resources, timelines, weakness of the program”). We used these terms to enhance inclusivity, as they are more familiar nomenclature with the case study discussants. In our analysis and the remainder of this report, we use the term “enabler,” which is conceptualized similarly to Mallen et al. [62] and defined as a factor that helped the project team conduct the evaluation. Similarly, we use the term “barrier” to mean an obstacle that makes evaluation less efficient or effective or result in additional delays or costs [62,63]. We completed several cycles of feedback and clarification before finalizing the case notes.

Data analysis

For this project, evaluation is defined as “the systematic collection of information about the activities, characteristics, and results of programs, to make judgments about the program, improve or further develop program effectiveness, inform decisions about future programming, or increase understanding” [64]. Once evaluations were identified, analysts abstracted and coded key attributes of each project. PM data was abstracted by compiling all the PM data into a single table and searching for specific phrases in the text. Search terms included the health hazard addressed, scope of the evaluation, evaluation purpose, design, and methods (none of these were mutually exclusive categories). Evaluations were not always described in these terms in the PMs and APRs; analysts additionally used descriptive information about the activities provided by the recipient to determine the most appropriate categorization.

The hazards addressed were based on pre-existing PM categories which included: all hazards (defined as extreme weather or climate-related hazard of any kind), heat, vector-borne disease, flooding/extreme precipitation, hurricanes, or wildfire and wildfire smoke. The scope of the evaluation was either action or project level, portfolio (about a set of related activities or subawards to other agencies doing similar projects) or the overall program (typically involving many disparate activities or projects). Evaluations were also categorized as formative, process, and/or outcome (see S1 Table). Each type involves the systematic collection of information. Formative evaluation occurs in the early stages of implementation to guide selecting, developing, tailoring, or improving an activity or program. Process evaluation aims to monitor implementation, often in terms of fidelity and reach, and support mid-course changes and improvements [65]. Outcome evaluation aims to assess the effectiveness, impact, or merit of a program to make recommendations about future program direction or improvement. Evaluation design categories were quasi-experimental, non-experimental strictly monitoring, or qualitative. For this project, quasi-experimental involved the collection of the same data at multiple time points or use of a comparison group (e.g., pre-post tests without a comparison group, pre-post tests with a nonequivalent comparison group or post-test only) [66]. Non-experimental involved data collection at a single time point or collection of different indicators over time (e.g., post-test only, cross-sectional, or case studies) [66]. Last, the methods were coded according to the reported data collection strategy in the PM or APR.

In some cases, the APR did not designate an activity as evaluation, however if a recipient reported a systematic process for collecting, analyzing or using information to achieve one of the three objectives associated with an evaluation type, the analyst documented it as an evaluation. Attributes of the evaluations were summarized using basic descriptive statistics [59–61]. The qualitative case notes for the three case studies were uploaded into a separate dataset for coding in the qualitative analysis software Dedoose (v9.0.46). Two members of the research team served as coders and a third managed the process for finalizing the codebook and adjudicating coding disagreements. The team used an inductive approach to develop an initial set of codes [67]. Both coders then independently coded samples of data and met regularly to reconcile coded content and update the code list and definitions. This process was repeated until the coders achieved a combined average Cohen’s Kappa above 0.70 for all of the codes, indicating “good agreement” [68]. After this point was reached, both coders finished coding the remaining qualitative data (i.e. double coded). An analyst performed content analysis to determine those themes which emerged most prominently [69]. To enable a better understanding of theme salience, coders quantified the frequency of theme mentions. The synthesis below is based on the presence of the theme within and across the three cases.

Ethics statement

CDC determined that the data collection was non-research, and no human subjects review was conducted in accordance with applicable federal law and CDC policy. Data was not collected from human research subjects to complete this project; informed consent was not applicable.

Review of CRSCI evaluations

Recipients reported a total of 76 distinct evaluations (Table 1). The evaluated projects or programs most commonly addressed all hazards or extreme weather (44.7%), followed by heat (27.6%). Most evaluations were conducted at the project level (75%), rather than for an entire portfolio (10.5%) or overall program (10.5%). Many evaluations had multiple purposes (23.7%), but the most common was to monitor implementation (59.2%) and assess whether the program was effective (42.1%). The most common type of design was non-experimental (44.7%), followed by quasi-experimental (27.6%). Surveys were the most common method used to collect data (56.6%).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Attributes of evaluations conducted by CRSCI recipients, 2016–2021.

https://doi.org/10.1371/journal.pclm.0000102.t001

Case studies

Case studies are summarized in Table 2 and more fully described below.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Summary of evaluation case studies, CRSCI recipients, 2016–2021.

https://doi.org/10.1371/journal.pclm.0000102.t002

1. Case Study 1: New York City’s Be a Buddy program for expanding social support during weather emergencies

Cross-cutting practices and themes from case studies

We noted several similarities and contrasts between the three case studies (Fig 1). These span several domains, including prioritization of health equity goals, inclusion of stakeholders in the evaluation process, diversity and local relevance of evaluation aims, and inconsistencies in using an evaluation framework.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Count of practices and cross-cutting themes from evaluation case studies, CRSCI recipients, 2016–2021.

https://doi.org/10.1371/journal.pclm.0000102.g001

Improving evaluation for CDC’s climate ready states and cities initiative and beyond

The review of evaluation activities across CRSCI indicates that evaluations were conducted by all recipients and focused on a range of adaptation initiatives. The majority of evaluations intended to monitor program implementation or improvement, and as expected based on the aims of the evaluations, non-experimental designs were most common. While some recipients also intended to assess the effectiveness of the adaptation intervention (42.1%), not all of those evaluations were designed to be able to answer outcome questions (none were experimental and only 27.6% were quasi-experimental, which could have involved a single group pre-post test without a comparison).

The case studies provided more insight as to why this might be the case. The grant stipulated recipients use the BRACE framework to plan, implement, and evaluate their adaptation interventions. However, in practice, recipients had difficulty stretching the funding award amount to cover the costs of robust evaluations in addition to planning and implementation activities, often translating into the inability to hire evaluation staff with the knowledge and skills required to mount more sophisticated evaluation designs.

In response to these challenges, evaluation was further emphasized and prioritized during the 2021–2026 cycle of CRSCI; recipients were required to conduct outcome evaluations of at least two adaptation interventions and to measure outcomes in terms of health equity. In this cycle, a modest increase in funding and a de-emphasis of other BRACE activities were intended to help facilitate this effort. Within the first year of the grant, two thirds have hired or plan to hire staff with evaluation training who will focus on these activities (internal administrative data, 2022). In addition, CHP has provided monthly evaluation training via a community of practice, evaluation resources, and templates to recipients. CHP also partnered with the American Public Health Association to publish a practical guide for justice and equity-focused climate adaptation and evaluation [73,74]. Beyond CRSCI, we recommend that adaptation funders provide adequate evaluation funding, supply training resources, use evaluation frameworks, and set clear expectations for locally driven evaluations that consider impacts among the most vulnerable, as well as conduct portfolio-level monitoring and evaluation that can reflect progress and, ultimately, the value of the overall program.

We noted that all three case studies included formative evaluation questions. In each instance, evaluation teams systematically sought input from populations intended to benefit from the intervention that would help inform design and implementation of the present intervention or the next iteration of interventions seeking the same outcome. Seeking to understand how a proposed intervention meaningfully responds to the community’s needs and context aligns with the stated aims to improve health outcomes among marginalized populations acutely vulnerable to climate hazards due to low adaptive capacity and high sensitivity. Formative evaluations also likely reflect that program planners were able to draw upon a limited evidence base that is largely unable to speak to the nuances of specific climate-hazard contexts. This highlights the vast information needs of programs such as these and suggests that evaluation will likely be pulling “double duty” for the foreseeable future. Double-loop learning is another way of conceptualizing learning for and from adaptation [75], in which evaluators routinely need to ask not only “How effective are we?” but also “What else could we be doing?”.

The case studies revealed a few additional barriers and enablers that correspond to those identified by prior research. Moser and Ekstrom [38] and Mallen et al. [62] similarly identified limitations in resources and funding, challenges with leadership, and lack of climate-specific expertise to be substantial barriers to successful climate adaptation. These findings imply that several barriers to successful adaptation also apply to evaluation. With two of three case studies noting lack of evaluation expertise as a significant barrier, lack of expertise may continue to present substantial challenges in this nascent field. The COVID-19 pandemic also presented significant barriers to evaluation progress for two of the three cases. This supports similar findings by Mallen et al. that the pandemic was one of the most common barriers to climate and health adaptation, impacting 10 out of 11 climate adaptation actions analyzed by the authors. We noted that enabling factors included the perceived need of the evaluation by the local team, especially among stakeholders in a position to provide resources, and the availability of both evaluation expertise and time that could be dedicated among existing staff. In all three cases, the evaluations were instigated and led by local teams who shared the perspective that learning about the program was of value. This aligns with a key principle of adaptive management, upon which the BRACE model for adaptation is built [7].

Use of evaluation frameworks

No evaluation teams featured in the case studies described seeking and using a published evaluation framework to guide planning and implementation. To varying degrees, all recipients followed a grant-required planning and reporting template, which was based on the CDC Evaluation Framework. We echo the calls of others that using a framework can be of great value [27,36], especially for evaluation teams without a professional evaluator. Multiple frameworks available and most offer step-by-step guidance along with training guides, reflective questions, and case studies [24,42–51]. Notably, few frameworks were created for practitioners in high-income countries [24,42,44,49] and fewer still explicitly address equity concerns [24,46,49]. There may also be a need to socialize the use of frameworks as standard practice and help practioners become more familiar and comfortable with using them. Health-focused climate adaptation in the US domestic setting could benefit from the development and promotion of a comprehensive yet flexible evaluation model.

There is growing evidence of maladaptation across all sectors, which can increase vulnerability and exacerbate existing inequalities [22,39]. We found that CRSCI evaluations did not generally address the risks of maladaptation. Evaluation should routinely prioritize conceptualizing and measuring these risks and could start with using theory of change or logic models to articulate how the adaptation will intervene on the fundamental drivers of climate vulnerability and health inequity [26,36,76,77]. Logic models can also prompt planners to consider the implementer’s sphere of control or authority and the range of potential outcomes outside of that sphere.

More specifically, it may prove useful for CRSCI and other practitioners to use rubrics or guides on the practical steps to make these considerations explicit in the planning process. Magnan et al [39] recommend three frameworks to preemptively and objectively assess maladaptation risks; however, none are specific to health-focused climate adaptation. Additional research and development of practical guidance will be crucial for crafting adaptation interventions and assessing their impact–intended and unintended, positive and negative.

Limitations and future directions

There are several limitations to note in this report. First, this study was based on grant recipient reporting via PMs and APRs. Our findings assume that all recipients reported relevant data accurately, comprehensively, and in a standardized fashion. We recognize that this may not be the case for all recipients via PM and APR reporting or the case studies. In some cases, reporting may be incomplete or not describe the extent to which an action or intervention was successful, as the PMs did not directly collect information about the effectiveness of an intervention or evaluation activities. In the APRs and PMs, grant recipients may have only reported what they deemed to be worthy or required of them to report to CDC and may not have reported all evaluation activities. As a result, our findings may be an underestimate of the total evaluation activities across all grant recipients.

Furthermore, in the case studies, we did not require that the evaluation itself be considered a “success” by pre-defined metrics or by the local team. This study seeks to identify methods to improve evaluation practice rather than identify successful interventions or evaluations, so defining success or success metrics was outside the scope of the current study. Second, our recommendations are based on the assumption that the experiences of these recipients are generalizable to other practitioners, which may not be accurate due to differences in funding and access to federally-provided technical assistance.

Third, qualitative findings were based on a limited set of case studies. Rather than statistical generalization, case studies rely on analytical generalization, which allows the user to apply a particular set of results to a broader theory [59]. The selection of multiple case studies aimed to meet a standard of replication and thereby expand the possible interpretation and application of the results [78]. There were also challenges during the grant cycle that impacted the data reporting strategy. Approval for performance measure collection was delayed, resulting in a one-year gap in reporting. In practice APRs were not entirely standardized across recipients, resulting in variability in the depth and details provided.

Evaluation, in the context of adaptation and health, is likely to grow in quality, scope, and scale over time, in response to the demands of funders for accountability and evidence of effectiveness, along with growing availability of data reflecting both climate vulnerability and resilience. With climate impacts on health increasing in frequency and severity, there is a growing need for rigorous evaluation of adaptation interventions to improve public health and justify public investment [34]. In tandem, evaluators may have more choices in terms of the evaluation questions, designs, and methods they may employ. In this context, it should become routine practice to explicitly understand and assert the inherent assumptions reflected in the choice of evaluation questions, design, and methods, as well as the evaluation stakeholders invited to the table. There are multiple ways of understanding effectiveness [18]. The evaluation questions highlighted in the case studies appear to be using two implicit frames for understanding and assessing effectiveness, namely “improved wellbeing” and “reduced vulnerability or increased adaptive capacity.” Adaptation success should be defined on a case-by-case basis, keeping local context in mind, and informed by the intervention stakeholders. However, for future iterations of CRSCI, evaluation teams may be encouraged to articulate what effectiveness frame is being applied and why. Further, CHP could also encourage the use of additional frames suggested by Singh et al [19], such as “enhanced resilience,” “sustainable adaptation,” “avoiding maladaptation,” “just and equitable adaptation,” or “transformative adaptation”.