Introduction

The 2005 International Health Regulations (IHR) provide an overarching framework to assist countries to “prevent, protect against, control and provide a public health response to the international spread of disease” and defines countries’ rights and obligations in handling emergencies that have the potential to cross borders. The Joint External Evaluation (JEE) is a voluntary, collaborative, multisectoral process led by national governments and supported by the World Health Organization (WHO) that aims to assess and monitor national capacities to implement IHR, with a view to identifying opportunities to strengthen preparedness and response.

Disease outbreaks are often unpredictable and require a range of approaches for preparedness and response that may or may not be adequately addressed by IHR and JEE. The unprecedented scale of the COVID-19 pandemic contrasts greatly with the more typical and much more frequent local epidemics of known diseases, such as cholera and yellow fever [1]. Given the severe impact COVID-19 has had in nations largely believed to be operating in line with IHR, and thus prepared to confront outbreaks, the relevance of IHR competencies and validity of the JEE have been questioned. At the request of Member States, WHO recently convened a Review Committee on the Functioning of IHR during COVID-19, concluding that much of what is contained within the current IHR is appropriate and well-considered, but was not sufficiently implemented prior to the pandemic [2].

The Global Health Security Index (GHSI) was developed in 2019, as an independent adjunct to the JEE. A 2020 regression analysis concluded that, for both the JEE and the GHSI, higher total scores were associated with fewer deaths from communicable diseases in general [3]. However, a separate analysis showed that these scores were not correlated with COVID-19 mortality across countries during the early phase of the pandemic [4], although the study failed to account for important confounders. A later regression analysis showed that higher JEE scores were correlated with fewer COVID-19 deaths [5], but adjusted for testing rates (likely to reflect the true numbers of cases) as a proxy for testing capacity, with only a very limited set of countries in multivariable analysis.

To improve pandemic preparedness and risk assessment, it is necessary to understand the extent to which the relationship between national scores on the JEE and deaths from COVID-19 differs from that for other communicable diseases. This analysis will support policy makers to review the utility of the JEE in assessing preparedness against both epidemics and pandemics.

Methods

Results

We identified and analyzed a total of 96 JEE reports. Data on COVID-19 deaths were unavailable for five countries (Cambodia, Eritrea, Federated States of Micronesia, Laos and Turkmenistan), leaving 91 for analysis (Table 1). High-income countries and the America and Europe region had the highest JEE scores, and low-income countries and the African region had the lowest scores. The median JEE score (as a percentage of the maximum possible score) across all countries was 50% (IQR 37.9–66.0%). Scores varied across JEE domains, with the highest scores for ‘detect’ (median 58.5, IQR 51.1–71.0), followed by ‘prevent’ (median 54.3, IQR 37.6–65.5) and ‘respond’ (median 43.7, IQR 30.6–66.4).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Joint External Evaluation scores by country-income group and geographical region (N = 96).

https://doi.org/10.1371/journal.pgph.0000246.t001

All of the investigated factors included in the multivariable model for COVID-19 deaths, were significantly predictive for deaths at all time points on univariable regression. Similarly, all of the factors included in the multivariable model investigating communicable disease deaths (2019) were significantly predictive in univariable models. Pre-pandemic tourism was not significantly associated with COVID-19 deaths on univariable regression, and was therefore excluded from multivariable models. The proportion of GDP spent on health was not significantly associated with communicable disease deaths on univariable regression and was therefore excluded from the multivariable model.

JEE score was positively associated with log COVID-19 deaths at 12 months into the pandemic but negatively associated with log communicable disease deaths (Fig 1). These relationships were statistically significant on univariable linear regression (S1 and S2 Tables). However, on multivariable linear regression (Table 2) the positive association between JEE scores and log COVID-19 deaths became non-significant at 6 and 12 months, while the negative association between JEE scores and log communicable disease deaths remained statistically significant (β 0.03, p = 0.003), with a 0.03 decrease in log communicable disease deaths for every percentage point increase in JEE score.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. COVID-19 deaths (10^th March 2021) and communicable disease deaths (2019) by country Joint External Evaluation scores.

https://doi.org/10.1371/journal.pgph.0000246.g001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Association between Joint External Evaluation scores and COVID-19 deaths per 100,000 population from multivariable linear regression models.

https://doi.org/10.1371/journal.pgph.0000246.t002

Higher test positivity rates, representing poor testing capacity or high levels of transmission, were associated with higher log COVID-19 deaths at all time points. Stringency Index was also associated with higher log COVID-19 deaths at all time points but adding test positivity rates into the model at least partially attenuated this positive association (S3–S5 Tables). While UHC indices (β -0.04 p<0.001) and international tourist arrivals were associated with communicable disease deaths (β 0.02, p = 0.002), they were not significantly associated with COVID-19 deaths.

Review of scatter plots of residuals against fitted values showed no violations of heteroskedasticity and quantile plots of residuals showed no departures from normality in any of the models. Sensitivity analysis showed that these results remained valid even after altering the variables included in each multivariable model (S3–S6 Tables).

The proportion of the variation in COVID-19 deaths at 12 months and communicable disease deaths explained by both models was high, with R² values of 0.60 and 0.75 respectively.

Discussion

Countries with higher JEE scores were associated with significantly fewer overall communicable disease deaths, but not COVID-19 deaths. This finding remained even after accounting for a range of other factors that could influence the distribution of COVID-19 deaths across countries and after considering multiple time points. This suggests that the JEE may be better suited to assessing epidemics of known diseases compared to a pandemic of a novel pathogen. Test positivity rates were the only factor strongly associated with more COVID-19 deaths at 12 months, further supporting the notion that the determinants of the impact of epidemics and pandemics may differ.

Epidemics of known diseases and pandemics of novel pathogens pose greatly different challenges when it comes to testing. The JEE explicitly assesses testing for infectious diseases through the availability of diagnostics, laboratory systems and real-time indicator and event-based surveillance systems [22]. It does not focus on scale, size and surge capacity of national infectious disease testing systems. Developing and scaling up the availability of tests, including RT-PCR and rapid tests such as the lateral flow assay, has been an unprecedented challenge for all countries during the pandemic [23]. Although the first diagnostic tests for COVID-19 were developed within 2 weeks of the reference genome being published [24], countries scrambled to compete for scarce global resources to expand testing programmes, leading to vast global inequities in access to testing. For example, in April 2020, while the USA was approaching 4 million tests conducted, Nigeria, Africa’s most populous country with a population almost two-thirds that of the USA, was nearing only 7000 tests [25]. In our multivariable analysis, even after accounting for JEE scores, weaker testing capacity (indicated by high test positivity rates) was significantly associated with increased COVID-19 deaths at 12 months. Given the existing but limited assessment of infectious disease testing capacities and scalability in the current JEE, this may be an area for early reform, if the JEE is to be used as an effective pandemic preparedness risk assessment tool.

Although the JEE is a self-assessment tool and was not designed to compare countries’ performance, the indicators should reflect necessary components of epidemic preparedness; logically, countries with higher scores should be better prepared to respond to a pandemic and limit transmission and mortality. The fact that JEE scores were not associated with fewer COVID-19 deaths goes against this expectation, but is in line with previous findings [4,5]. Possible reasons for the discrepancy in associations between JEE scores and COVID-19 deaths compared to other communicable disease deaths include 1) For many common infections, the natural history of disease is well understood, clinical tests are available and surveillance systems are well-established. In the early stages of the COVID-19 pandemic, global testing capacity was inadequate, including in many high-income country global travel hubs, which made countries vulnerable to imported cases. Consequently, the true burden of COVID-19 cases in real-time was likely underestimated [26], limiting effective public health action; 2) The pandemic required outbreak response capacities of a scale not seen for smaller epidemics of known diseases. Existing infrastructure for testing, contact tracing, infection control and clinical management had to be rapidly scaled up and reorganized [27,28]; 3) In the absence of vaccines or therapeutics, few disease control measures for COVID-19 were initially available, as they might be for other common epidemic-prone diseases, such as tuberculosis or measles. In this context, individual behaviours across populations, including physical distancing and mask wearing, became an important determinant of spread [29], in turn affected by public risk communication as well as individual attitudes toward infectious diseases, personal risk and freedoms; 4) Unlike for small outbreaks of known diseases, decision-making on control measures took place largely outside of traditional public health agencies, with political leaders having to weigh trade-offs in a highly public time-pressured environment and with significant scientific uncertainty; and 5) In its current form, the JEE considers a range of factors but focuses on the existence of capacities, such as policies, procedures, and systems, rather than the quality of them, which is more difficult to demonstrate, and the ability to deliver these capacities at the speed and scale required in a pandemic. Other limitations of the JEE, such as potential bias created from self-assessment and limited scope [3] may further help explain the disparity in the relationship between communicable disease and COVID-19 deaths.

Limitations of our study include: 1) Our study may be vulnerable to selection bias given that not all countries have volunteered for the JEE (analysable data were available only from 91 countries, less than half of the world’s total), and those that have may be different to those that have not; 2) The JEE was not designed as a comparative tool across settings but rather to allow countries to identify areas of weakness and track progress. JEE implementation and interpretation may therefore vary across different countries and over time, independent of other factors; 3) We used COVID-19 deaths in our analysis, which may not be recorded in the same way across countries, although these data are likely less susceptible to biases related to testing and healthcare-seeking compared to some other indicators, such as numbers of cases; 4) Underreporting of deaths due to inadequate mortality surveillance, if associated with lower JEE scores, could have biased the results. We considered open-access data on excess mortality [30] as an alternative and more comprehensive way of measuring COVID-19 impact but, due to large amounts of missing and poor data quality, did not pursue this approach; and 5) Our study covered only a 12-month period during which vaccine distribution has been highly unequal across the world [31]. The associations between JEE scores and COVID-19 deaths may change as the pandemic progresses.

Our findings have implications for global health policy. Many of the countries with the highest numbers of cases and deaths from COVID-19 are also considered to have the most robust IHR capacities and health systems as measured by the JEE. The IHR, and the JEE that is derived from it, may therefore require design change or supplementation through other mechanisms if the wide range of factors involved in preventing, detecting and responding to both future pandemics and epidemics of known diseases are to be comprehensively captured through country risk assessments. It may be necessary to develop and test more complex and demanding requirements within tools like the JEE, simulation exercises and after-action reviews, to improve future emergency preparedness by better measuring real-world capabilities. For example, approaches could better account for the altered response capacities, disease control measures, and political considerations that may be necessary in dealing with pandemics and novel pathogens. Our results also underscore the importance of testing capacity, which may prove a sensible early target for the pandemic-focussed reform of existing tools.

Given that the existing public health response capacities outlined in Annex 1 of the IHR [32] are still not adequately implemented in many countries, including more requirements within the JEE may not translate into better emergency preparedness. Indeed, one critical implication from our findings is the need to recognize the limitations in the utility of JEE as a measure of pandemic preparedness, given the uncertainty of how any future pandemic may look and the potentially devastating consequences of complacency or overconfidence in response capabilities. Various factors may improve the implementation of IHR and deserve attention alongside the scrutiny of risk assessment tools. These include stronger public health systems, workforce and institutions; financial incentives; appropriate decentralisation of powers; community and private sector engagement; upskilling the public health workforce; improved data collection, transparency and sharing; supportive legal instruments; national leadership; and regional collaboration [33,34].

Much of the recent discourse on global health security has emphasised integration with the One Health, UHC and health system strengthening agendas [35–37], underscoring the links between humans, animals, ecosystems and health systems. The discussion on pandemic preparedness must expand to consider essential public health functions [38], providing an opportunity to consider pandemic risk and preparedness in the broader context of actions required to promote and protect health. High quality test, trace and isolate systems (often operating outside the traditional structures of health systems) have proved an essential part of early pandemic response in many countries [39]. Nor can the focus be entirely on the implicated pandemic pathogen; during the first peak in the United Kingdom (UK), 96% of COVID-19 deaths were in individuals with at least one pre-existing medical condition, most of which are chronic non-communicable diseases [40]. Those living in more densely populated and deprived areas, working in high-risk occupations with poor access to healthcare, are at a high risk of infection and severe illness [41,42]. Sociocultural and political factors have also been found to be important predictors of COVID-19 deaths across countries [12,43,44] compared to smaller-scale outbreaks of known infectious diseases. Capturing such vulnerabilities as part of assessments of preparedness will require more research and an expansion in ambition, but may help to improve the utility of risk assessments. Effective systems of public sector governance will be required to operationalize essential public health functions. Pandemic preparedness assessments must also build an understanding of how evidence is used in decision-making, and how well-informed, fair, and inclusive strategic decisions are likely to be in emergency scenarios, regardless of more specific public health vulnerabilities or capacities.

Supporting information