Does it matter that standard preparedness indices did not predict COVID-19 outcomes?

A number of scientific publications and commentaries have suggested that standard preparedness indices such as the Global Health Security Index (GHSI) and Joint External Evaluation (JEE) scores did not predict COVID-19 outcomes. To some, the failure of these metrics to be predictive demonstrates the need for a fundamental reassessment which better aligns preparedness measurement with operational capacities in real-world stress situations, including the points at which coordination structures and decision-making may fail. There are, however, several reasons why these instruments should not be so easily rejected as preparedness measures. From a methodological point of view, these studies use relatively simple outcome measures, mostly based on cumulative numbers of cases and deaths at a fixed point of time. A country’s “success” in dealing with the pandemic is highly multidimensional – both in the health outcomes and type and timing of interventions and policies – is too complex to represent with a single number. In addition, the comparability of mortality data over time and among jurisdictions is questionable due to highly variable completeness and representativeness. Furthermore, the analyses use a cross-sectional design, which is poorly suited for evaluating the impact of interventions, especially for COVID-19. Conceptually, a major reason that current preparedness measures fail to predict pandemic outcomes is that they do not adequately capture variations in the presence of effective political leadership needed to activate and implement existing system, instill confidence in the government’s response; or background levels of interpersonal trust and trust in government institutions and country ability needed to mount fast and adaptable responses. These factors are crucial; capacity alone is insufficient if that capacity is not effectively leveraged. However, preparedness metrics are intended to identify gaps that countries must fill. As important as effective political leadership and trust in institutions, countries cannot be held accountable to one another for having good political leadership or trust in institutions. Therefore, JEE scores, the GHSI, and similar metrics can be useful tools for identifying critical gaps in capacities and capabilities that are necessary but not sufficient for an effective pandemic response.