Measuring causes of death in populations: a new metric that corrects cause-specific mortality fractions for chance

BACKGROUND: Verbal autopsy is gaining increasing acceptance as a method for determining the underlying cause of death when the cause of death given on death certificates is unavailable or unreliable, and there are now a number of alternative approaches for mapping from verbal autopsy interviews to the underlying cause of death. For public health applications, the population-level aggregates of the underlying causes are of primary interest, expressed as the cause-specific mortality fractions (CSMFs) for a mutually exclusive, collectively exhaustive cause list. Until now, CSMF Accuracy is the primary metric that has been used for measuring the quality of CSMF estimation methods. Although it allows for relative comparisons of alternative methods, CSMF Accuracy provides misleading numbers in absolute terms, because even random allocation of underlying causes yields relatively high CSMF accuracy. Therefore, the objective of this study was to develop and test a measure of CSMF that corrects this problem. METHODS: We developed a baseline approach of random allocation and measured its performance analytically and through Monte Carlo simulation. We used this to develop a new metric of population-level estimation accuracy, the Chance Corrected CSMF Accuracy (CCCSMF Accuracy), which has value near zero for random guessing, and negative quality values for estimation methods that are worse than random at the population level. RESULTS: The CCCSMF Accuracy formula was found to be CCSMF Accuracy = (CSMF Accuracy - 0.632) / (1 - 0.632), which indicates that, at the population-level, some existing and commonly used VA methods perform worse than random guessing. CONCLUSIONS: CCCSMF Accuracy should be used instead of CSMF Accuracy when assessing VA estimation methods because it provides a more easily interpreted measure of the quality of population-level estimates. ELECTRONIC SUPPLEMENTARY MATERIAL: The online version of this article (doi:10.1186/s12963-015-0061-1) contains supplementary material, which is available to authorized users.