Cargando…

Validating hierarchical verbal autopsy expert algorithms in a large data set with known causes of death

BACKGROUND: Physician assessment historically has been the most common method of analyzing verbal autopsy (VA) data. Recently, the World Health Organization endorsed two automated methods, Tariff 2.0 and InterVA–4, which promise greater objectivity and lower cost. A disadvantage of the Tariff method...

Descripción completa

Detalles Bibliográficos
Autores principales: Kalter, Henry D, Perin, Jamie, Black, Robert E
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Edinburgh University Global Health Society 2016
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4766791/
https://www.ncbi.nlm.nih.gov/pubmed/26953965
http://dx.doi.org/10.7189/jogh.06.010601
_version_ 1782417733065900032
author Kalter, Henry D
Perin, Jamie
Black, Robert E
author_facet Kalter, Henry D
Perin, Jamie
Black, Robert E
author_sort Kalter, Henry D
collection PubMed
description BACKGROUND: Physician assessment historically has been the most common method of analyzing verbal autopsy (VA) data. Recently, the World Health Organization endorsed two automated methods, Tariff 2.0 and InterVA–4, which promise greater objectivity and lower cost. A disadvantage of the Tariff method is that it requires a training data set from a prior validation study, while InterVA relies on clinically specified conditional probabilities. We undertook to validate the hierarchical expert algorithm analysis of VA data, an automated, intuitive, deterministic method that does not require a training data set. METHODS: Using Population Health Metrics Research Consortium study hospital source data, we compared the primary causes of 1629 neonatal and 1456 1–59 month–old child deaths from VA expert algorithms arranged in a hierarchy to their reference standard causes. The expert algorithms were held constant, while five prior and one new “compromise” neonatal hierarchy, and three former child hierarchies were tested. For each comparison, the reference standard data were resampled 1000 times within the range of cause–specific mortality fractions (CSMF) for one of three approximated community scenarios in the 2013 WHO global causes of death, plus one random mortality cause proportions scenario. We utilized CSMF accuracy to assess overall population–level validity, and the absolute difference between VA and reference standard CSMFs to examine particular causes. Chance–corrected concordance (CCC) and Cohen’s kappa were used to evaluate individual–level cause assignment. RESULTS: Overall CSMF accuracy for the best–performing expert algorithm hierarchy was 0.80 (range 0.57–0.96) for neonatal deaths and 0.76 (0.50–0.97) for child deaths. Performance for particular causes of death varied, with fairly flat estimated CSMF over a range of reference values for several causes. Performance at the individual diagnosis level was also less favorable than that for overall CSMF (neonatal: best CCC = 0.23, range 0.16–0.33; best kappa = 0.29, 0.23–0.35; child: best CCC = 0.40, 0.19–0.45; best kappa = 0.29, 0.07–0.35). CONCLUSIONS: Expert algorithms in a hierarchy offer an accessible, automated method for assigning VA causes of death. Overall population–level accuracy is similar to that of more complex machine learning methods, but without need for a training data set from a prior validation study.
format Online
Article
Text
id pubmed-4766791
institution National Center for Biotechnology Information
language English
publishDate 2016
publisher Edinburgh University Global Health Society
record_format MEDLINE/PubMed
spelling pubmed-47667912016-03-07 Validating hierarchical verbal autopsy expert algorithms in a large data set with known causes of death Kalter, Henry D Perin, Jamie Black, Robert E J Glob Health Research Theme: Verbal/Social Autopsy BACKGROUND: Physician assessment historically has been the most common method of analyzing verbal autopsy (VA) data. Recently, the World Health Organization endorsed two automated methods, Tariff 2.0 and InterVA–4, which promise greater objectivity and lower cost. A disadvantage of the Tariff method is that it requires a training data set from a prior validation study, while InterVA relies on clinically specified conditional probabilities. We undertook to validate the hierarchical expert algorithm analysis of VA data, an automated, intuitive, deterministic method that does not require a training data set. METHODS: Using Population Health Metrics Research Consortium study hospital source data, we compared the primary causes of 1629 neonatal and 1456 1–59 month–old child deaths from VA expert algorithms arranged in a hierarchy to their reference standard causes. The expert algorithms were held constant, while five prior and one new “compromise” neonatal hierarchy, and three former child hierarchies were tested. For each comparison, the reference standard data were resampled 1000 times within the range of cause–specific mortality fractions (CSMF) for one of three approximated community scenarios in the 2013 WHO global causes of death, plus one random mortality cause proportions scenario. We utilized CSMF accuracy to assess overall population–level validity, and the absolute difference between VA and reference standard CSMFs to examine particular causes. Chance–corrected concordance (CCC) and Cohen’s kappa were used to evaluate individual–level cause assignment. RESULTS: Overall CSMF accuracy for the best–performing expert algorithm hierarchy was 0.80 (range 0.57–0.96) for neonatal deaths and 0.76 (0.50–0.97) for child deaths. Performance for particular causes of death varied, with fairly flat estimated CSMF over a range of reference values for several causes. Performance at the individual diagnosis level was also less favorable than that for overall CSMF (neonatal: best CCC = 0.23, range 0.16–0.33; best kappa = 0.29, 0.23–0.35; child: best CCC = 0.40, 0.19–0.45; best kappa = 0.29, 0.07–0.35). CONCLUSIONS: Expert algorithms in a hierarchy offer an accessible, automated method for assigning VA causes of death. Overall population–level accuracy is similar to that of more complex machine learning methods, but without need for a training data set from a prior validation study. Edinburgh University Global Health Society 2016-06 2016-02-20 /pmc/articles/PMC4766791/ /pubmed/26953965 http://dx.doi.org/10.7189/jogh.06.010601 Text en Copyright © 2016 by the Journal of Global Health. All rights reserved. http://creativecommons.org/licenses/by/2.5/ This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.
spellingShingle Research Theme: Verbal/Social Autopsy
Kalter, Henry D
Perin, Jamie
Black, Robert E
Validating hierarchical verbal autopsy expert algorithms in a large data set with known causes of death
title Validating hierarchical verbal autopsy expert algorithms in a large data set with known causes of death
title_full Validating hierarchical verbal autopsy expert algorithms in a large data set with known causes of death
title_fullStr Validating hierarchical verbal autopsy expert algorithms in a large data set with known causes of death
title_full_unstemmed Validating hierarchical verbal autopsy expert algorithms in a large data set with known causes of death
title_short Validating hierarchical verbal autopsy expert algorithms in a large data set with known causes of death
title_sort validating hierarchical verbal autopsy expert algorithms in a large data set with known causes of death
topic Research Theme: Verbal/Social Autopsy
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4766791/
https://www.ncbi.nlm.nih.gov/pubmed/26953965
http://dx.doi.org/10.7189/jogh.06.010601
work_keys_str_mv AT kalterhenryd validatinghierarchicalverbalautopsyexpertalgorithmsinalargedatasetwithknowncausesofdeath
AT perinjamie validatinghierarchicalverbalautopsyexpertalgorithmsinalargedatasetwithknowncausesofdeath
AT blackroberte validatinghierarchicalverbalautopsyexpertalgorithmsinalargedatasetwithknowncausesofdeath