How comparable is COVID-19 mortality across countries?

Cross-Country Analysis


How comparable is COVID-19 mortality across countries?

Dashboards from the World Health Organization (WHO)Johns Hopkins University, and Worldometer, among others, publish information on daily numbers and rates of COVID-19 cases and deaths. However, they rely on summary data supplied by countries (or, sometimes, a variety of sources) and the deaths ultimately counted as COVID-19 can differ substantially, complicating cross-country comparisons. 

Surveillance and monitoring systems are at the core of governments’ responses to the COVID-19 pandemic. This policy snapshot for the COVID-19 Health Systems Response Monitor (HSRM) Analysis section focuses on how COVID-19 mortality is recorded to facilitate comparative monitoring and reporting.  

How are COVID-19 deaths defined? 

Headline figures for COVID-19 (those reported daily by the official Government sources for the current or previous day) of cases and deaths have a benefit of being real-time or near real-time. It is important to distinguish those from figures reported by the national statistical offices (or other agencies/authorities) that rely on more comprehensive death registers, but take longer to be processed. For a number of reasons (please see below), deaths reported daily in headline figures may not be entirely comparable across countries. 

Table 1 shows the definitions that are applied to COVID-19 deaths reported in the headline figures. There are two main ways in which COVID-19 deaths are defined. The first, based on the WHO definition (see below), uses clinically confirmed or probable COVID-19 case (e.g. Belgium, Canada, France, Germany) and is not dependant on the availability of a laboratory test. The second, on the other hand, is reliant primarily on a positive laboratory test (e.g. Austria, Italy, the Netherlands, Spain, the United Kingdom). At the same time, there is still important variation within these two groups, as there are countries which include probable COVID-19 deaths in the definition, but still in practice require a laboratory confirmation (e.g. Cyprus, Greece, Romania, Serbia), while there are also countries that primarily use clinical diagnosis, but also include any death among positive cases (e.g. Canada). Occasionally, countries make the distinction between deaths with COVID-19 and deaths from COVID-19 in the headline figures (e.g. Lithuania).  

Table 1. Definition of COVID-19 deaths in headline figures 

CountryClinical diagnosis-based (WHO definition) (confirmed and probable)Test-based (+ve test required)Other issues affecting comparisons
BelgiumVOnly lab-confirmed deaths (largely in hospital) reported until 31st March
BulgariaVHowever all reported deaths had +ve test result
CanadaVFigures include deaths from other causes “with” COVID
CroatiaVHowever, if considered ‘probable’ only included if test +ve
CyprusVVTest result has to be recent
EstoniaV
FranceV
GermanyV– figures include deaths “with” COVID; – possibly overestimate
GreeceVV
IrelandVOnly lab-confirmed deaths were reported until 21st April, but all subsequent figures also include probable deaths from the start of the pandemic
IsraelV
LatviaVAll reported deaths had +ve test result
LithuaniaV
MaltaVHowever, if considered ‘probable’ only included if test +ve
PolandV
PortugalVProbable deaths tested for COVID-19
RomaniaVV
SerbiaVV
USAVOnly lab-confirmed deaths reported until mid-April
AustriaV
Bosnia and HerzegovinaV
Hungary V 
IcelandV
ItalyV– due to requirement for +ve test, largely hospital deaths are included; – likely underestimate as alternative sources (e.g. statistical office) report higher numbers
The NetherlandsV– due to +ve test requirement, largely hospital deaths are included; – likely underestimate as alternative sources (e.g. statistical office) report higher numbers
NorwayV
Slovenia V– wide testing performed with all patients with moderate/severe respiratory symptoms hospitalised and tested
SpainV– due to +ve test requirement, largely hospital deaths are included; – likely underestimate as alternative sources (e.g. statistical office) report higher numbers
Switzerland V– may differ from data reported by cantons, where deaths also include clinical diagnosis
The United KingdomV– until 29th April only hospital deaths were included for England; – likely underestimate as alternative sources (Office for National Statistics, which publishes weekly data) report higher numbers

NB: these data are based on country expert opinion collated within HSRM initiative as of 31st May 2020. These are based on publicly available information and are subject to change.  

On 16th April 2020, WHO issued guidelines on certification and coding of COVID-19 as a cause of death.[1] Death due to COVID-19 is defined as: 

a death resulting from a clinically compatible illness, in a probable or confirmed COVID-19 case, unless there is a clear alternative cause of death that cannot be related to COVID disease (e.g. trauma). There should be no period of complete recovery from COVID-19 between illness and death. A death due to COVID-19 may not be attributed to another disease (e.g. cancer) and should be counted independently of pre-existing conditions that are suspected of triggering a severe course of COVID-19.” 

This definition requires COVID-19 to be identified on a death certificate by a clinician as an underlying cause where the disease caused, or is assumed to have caused, or contributed to death. It served as a basis for many national diagnosis-based definitions (Column 2 in Table 1). However, there may be variable delays in reporting due to the length of death certification process.  

The other definition relies on a positive test, and as a consequence, on rigorous testing policies and accuracy of available tests. As a result, the following considerations are needed:

  • Testing policies vary widely across countries; moreover, they tend to evolve over the course of the pandemic.[2] With the emergence of community spread and limited testing capacity, population groups eligible for tests in some countries were restricted (e.g. to people with severe symptoms). This resulted in limiting reporting to mainly hospital deaths (e.g. Italy, the Netherlands, Spain, the United Kingdom (England) as those attributed to COVID-19. At the same time, deaths in long-term care institutions and residential setting have been underreported.[3]
  • In addition to absolute number of deaths, the case-fatality ratio for COVID-19 is also affected by testing. Countries with very narrow testing criteria, for example those only testing severe cases that present in hospital, are likely to have comparatively high case-fatality rates as a result of the smaller volume of tests. 
  • In terms of testing accuracy, PCR test sensitivity can be as low as 54%,[4] with results also depending on the timeliness and expertise of sample collection. This means that a number of cases were not detected due to false negatives. 

Implications for interpreting the headline COVID-19 mortality figures

Given the variation in defining the COVID-19 death in the headline figures, caution in needed when making comparisons of COVID-19 mortality across countries. Where the WHO definition is used, it is more likely that a greater share of COVID-19-associated deaths will be captured, unlike in countries relying solely on positive tests – for reasons mentioned above. However, there are further caveats: recording of cause of death on the death certificates can vary due to the practical implementation in international and national guidelines, as well as death certification and coding practices. For example, some countries using the WHO definition still require a positive test result (e.g. Greece), while others (e.g. Canada) include any death in a person with COVID-19, even if it was not triggered by the virus (e.g. trauma).  There may also be changes in guidelines over time, which is particularly relevant during this pandemic, as it involves the emergence on a novel cause of death.    

Monitoring excess deaths can more accurately highlight the scale of COVID-19 impact

The issues mentioned above limit the comparability of the headline COVID-19 mortality figures between countries. Therefore, the European Centre for Disease Prevention and Control (ECDC) endorses the WHO definition and also recommends European countries monitor total, as well as excess mortality by age at least on a weekly basis.[5] Tracking all deaths has several advantages. Most importantly, it includes deaths among those who probably had COVID-19. It also provides a more comprehensive picture of the scale of mortality during the crisis and facilitates comparisons across countries. 

A number of initiatives of international comparisons using excess deaths have been developed. For example, the analysis done by the Financial Times on 28th May 2020 shows that there is an increase in mortality in comparison to levels of previous years of over 40% in a number of countries (e.g. in Italy, Spain, the Netherlands, Belgium, the United Kingdom).[6] These shares, as well as the overall increase in excess deaths, however, varied markedly across regions within countries. It is also important to bear in mind that impact varies by age group, sex and ethnicity, with higher mortality rates in men compared to women, and in older age groups as well as among Black, Asian and minority ethnic groups.[7]

The analysis done by the Economist shows that countries where reporting of COVID-19 mortality is linked to the requirement to have a positive test result show a comparatively lower proportion of total excess deaths are coded as COVID-19 deaths. Thus, as of 27th May, only half of excess deaths in Italy, 60% in the Netherlands and 77% in the United Kingdom were attributed to COVID-19 (Figure 1). In contrast, in Belgium, where the approach to defining COVID-19 deaths has been broader and reflects the WHO definition, the number of COVID-19 deaths appeared to be higher than total excess deaths, while in France it reaches 95%.[8]  In Spain, however, 90% of excess deaths has been attributed to COVID-19, suggesting that scale of under-reporting in countries relying on testing may differ across countries.  

Figure 1. COVID-19 deaths as % of total excess deaths in selected countries 

Source: The Economist (8)

Excess mortality not only makes it possible to better understand the overall impact of COVID-19 on population health, it also facilitates tracking the impact of the pandemic in real time, if reported with a minimal delay and at least on a weekly basis. This shows the important role of Statistical Offices or equivalent agencies in timely collection and publication of all-cause mortality data. As the analysis by the Health Foundation shows, during the peak of the pandemic, the number of deaths in Spain, Italy and the United Kingdom has more than doubled in comparison to the average figure for the corresponding week in the preceding 5 years.[9] A recent study from Sweden shows that from the first week of April onwards the country experienced an increase in excess mortality among people over 60 years of age, with those over 80 being particularly affected with a 75% increase in mortality in men and 50% in women.[10] That study also finds that this is leading to a rapid drop in life expectancy at age 50 – by 3 years in men and 2 years in women. Unfortunately, however, these crucial data are not routinely reported or tracked via dashboards in the same way as COVID-19 headline figures in most countries, even within the European Union. 

The exception is a subset of countries (18 European Union/European Economic Area (EU/EEA) countries, 2 regions of Germany and the 4 countries of the United Kingdom) whose agencies contribute to the EuroMOMO project.[11] Despite the approximately 4 weeks delay in publishing a complete dataset and the scale of excess deaths for individual countries or regions being expressed as a z score[12] (with each z unit being one standard deviation) rather than the more intuitive figure of the percentage excess mortality (or ideally, the actual figures to allow for more detailed inspection), it still shows that in April 2020 mortality in Belgium, France, Ireland, Italy, the Netherlands, Portugal, Spain, Sweden, Switzerland and the 4 countries of the United Kingdom was significantly higher than the levels seen between 2015 and 2019 for all ages. The highest rise in deaths were seen in people over 65 years of age, but high excesses (z > 7) were also seen in France, Spain and particularly in England among 15-64 year olds. 

Conclusions

In summary, national definitions of COVID-19 death fall broadly into two groups: diagnosis-based (confirmed and probable, in line with the WHO definition) and test-based. This may result in a substantial lack of comparability of COVID-19 related mortality across countries. In addition, issues such as testing policies, settings captured, change over time and regional variations in practices can further complicate mortality monitoring. 

Where headline figures are subject to laboratory test confirmation, there often is evidence from statistical offices or research agencies of substantial under-reporting of COVID-19 mortality. In contrast, figures that are based on death certificates are recognised as more reliable by both WHO and ECDC. These, however, take longer to be registered, and are more challenging to report promptly. In addition, where the WHO definition is used, accuracy may vary depending on the implementation of the WHO guidelines in practice within countries.

Estimating excess deaths could be used more widely to monitor the true scale of impact of the COVID-19 pandemic with minimal time lag. Early evidence already shows close to two-fold increase in excess mortality in countries most affected, resulting in years of life expectancy being wiped out. It is important for countries to put more effort into timely monitoring of the COVID-19 impact on mortality using a variety of data sources, and to publish these data in a timely and accessible manner.   

Marina Karanikolos, Martin McKee


[1] World Health Organization (16/04/2020) https://www.who.int/classifications/icd/Guidelines_Cause_of_Death_COVID-19.pdf?ua=1

[2] Health Systems Response Monitor (HSRM) (16/04/2020)  https://analysis.covid19healthsystem.org/index.php/2020/04/16/how-do-covid-19-testing-criteria-differ-across-countries/

[3] Health Foundation (13/05/2020) https://www.health.org.uk/news-and-comment/charts-and-infographics/deaths-from-any-cause-in-care-homes-have-increased

[4] European Commission (16/04/2020) https://ec.europa.eu/docsroom/documents/40805

[5] European Centre for Disease Prevention and Control (23/04/2020) https://www.ecdc.europa.eu/sites/default/files/documents/covid-19-rapid-risk-assessment-coronavirus-disease-2019-ninth-update-23-april-2020.pdf

[6] Financial Times (29/05/2020) https://www.ft.com/content/6b4c784e-c259-4ca4-9a82-648ffde71bf0

[7] Pan D, et al. (03/06/2020) The impact of ethnicity on clinical outcomes in COVID-19: A systematic review. The Lancet. DOI https://doi.org/10.1016/j.eclinm.2020.100404

[8] The Economist (27/05/2020) https://www.economist.com/graphic-detail/2020/04/16/tracking-covid-19-excess-deaths-across-countries

[9] Health Foundation (06/05/2020) https://www.health.org.uk/news-and-comment/charts-and-infographics/understanding-excess-mortality-the-fairest-way-to-make-international-comparisons

[10] Modig K, Ebeling M (2020) Excess mortality from COVID-19. Weekly excess rate deaths by age and sex for Sweden. MedRxiv preprint. DOI: https://doi.org/10.1101/2020.05.10.20096909

[11] Euromomo (accessed 14/05/2020) https://www.euromomo.eu/graphs-and-maps/

[12] Euromomo (accessed 14/05/2020) What is a z-score?  https://www.euromomo.eu/how-it-works/what-is-a-z-score/