Cargando…
Importance of external validation and subgroup analysis of artificial intelligence in the detection of low ejection fraction from electrocardiograms
AIM: Left ventricular systolic dysfunction (LVSD) carries an increased risk for overt heart failure and mortality, yet treatable to mitigate disease progression. An artificial intelligence (AI)-enabled 12-lead electrocardiogram (ECG) model demonstrated promise in LVSD screening, but the performance...
Autores principales: | , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Oxford University Press
2022
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9779862/ https://www.ncbi.nlm.nih.gov/pubmed/36710903 http://dx.doi.org/10.1093/ehjdh/ztac065 |
_version_ | 1784856714095362048 |
---|---|
author | Yagi, Ryuichiro Goto, Shinichi Katsumata, Yoshinori MacRae, Calum A Deo, Rahul C |
author_facet | Yagi, Ryuichiro Goto, Shinichi Katsumata, Yoshinori MacRae, Calum A Deo, Rahul C |
author_sort | Yagi, Ryuichiro |
collection | PubMed |
description | AIM: Left ventricular systolic dysfunction (LVSD) carries an increased risk for overt heart failure and mortality, yet treatable to mitigate disease progression. An artificial intelligence (AI)-enabled 12-lead electrocardiogram (ECG) model demonstrated promise in LVSD screening, but the performance dropped unexpectedly in external validation. We thus sought to train de novo models for LVSD detection and investigated their performance across multiple institutions and across a broader set of patient strata. METHODS AND RESULTS: ECGs taken within 14 days of an echocardiogram were obtained from four academic hospitals (three in the United States and one in Japan). Four AI models were trained to detect patients with ejection fraction (EF) <40% using ECGs from each of the four institutions. All the models were then evaluated on the held-out test data set from the same institution and data from the three external institutions. Subgroup analyses stratified by patient characteristics and common ECG abnormalities were performed. A total of 221 846 ECGs were identified from the 4 institutions. While the Brigham and Women’s Hospital (BWH)-trained and Keio-trained models yielded similar accuracy on their internal test data [area under the receiver operating curve (AUROC) 0.913 and 0.914, respectively], external validity was worse for the Keio-trained model (AUROC: 0.905–0.915 for BWH trained and 0.849–0.877 for Keio-trained model). Although ECG abnormalities including atrial fibrillation, left bundle branch block, and paced rhythm-reduced detection, the models performed robustly across patient characteristics and other ECG features. CONCLUSION: While using the same model architecture, different data sets produced models with different performances for detecting low-EF highlighting the importance of external validation and extensive stratification analysis. |
format | Online Article Text |
id | pubmed-9779862 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2022 |
publisher | Oxford University Press |
record_format | MEDLINE/PubMed |
spelling | pubmed-97798622023-01-27 Importance of external validation and subgroup analysis of artificial intelligence in the detection of low ejection fraction from electrocardiograms Yagi, Ryuichiro Goto, Shinichi Katsumata, Yoshinori MacRae, Calum A Deo, Rahul C Eur Heart J Digit Health Short Report AIM: Left ventricular systolic dysfunction (LVSD) carries an increased risk for overt heart failure and mortality, yet treatable to mitigate disease progression. An artificial intelligence (AI)-enabled 12-lead electrocardiogram (ECG) model demonstrated promise in LVSD screening, but the performance dropped unexpectedly in external validation. We thus sought to train de novo models for LVSD detection and investigated their performance across multiple institutions and across a broader set of patient strata. METHODS AND RESULTS: ECGs taken within 14 days of an echocardiogram were obtained from four academic hospitals (three in the United States and one in Japan). Four AI models were trained to detect patients with ejection fraction (EF) <40% using ECGs from each of the four institutions. All the models were then evaluated on the held-out test data set from the same institution and data from the three external institutions. Subgroup analyses stratified by patient characteristics and common ECG abnormalities were performed. A total of 221 846 ECGs were identified from the 4 institutions. While the Brigham and Women’s Hospital (BWH)-trained and Keio-trained models yielded similar accuracy on their internal test data [area under the receiver operating curve (AUROC) 0.913 and 0.914, respectively], external validity was worse for the Keio-trained model (AUROC: 0.905–0.915 for BWH trained and 0.849–0.877 for Keio-trained model). Although ECG abnormalities including atrial fibrillation, left bundle branch block, and paced rhythm-reduced detection, the models performed robustly across patient characteristics and other ECG features. CONCLUSION: While using the same model architecture, different data sets produced models with different performances for detecting low-EF highlighting the importance of external validation and extensive stratification analysis. Oxford University Press 2022-11-02 /pmc/articles/PMC9779862/ /pubmed/36710903 http://dx.doi.org/10.1093/ehjdh/ztac065 Text en © The Author(s) 2022. Published by Oxford University Press on behalf of the European Society of Cardiology. https://creativecommons.org/licenses/by-nc/4.0/This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial License (https://creativecommons.org/licenses/by-nc/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited. For commercial re-use, please contact journals.permissions@oup.com |
spellingShingle | Short Report Yagi, Ryuichiro Goto, Shinichi Katsumata, Yoshinori MacRae, Calum A Deo, Rahul C Importance of external validation and subgroup analysis of artificial intelligence in the detection of low ejection fraction from electrocardiograms |
title | Importance of external validation and subgroup analysis of artificial intelligence in the detection of low ejection fraction from electrocardiograms |
title_full | Importance of external validation and subgroup analysis of artificial intelligence in the detection of low ejection fraction from electrocardiograms |
title_fullStr | Importance of external validation and subgroup analysis of artificial intelligence in the detection of low ejection fraction from electrocardiograms |
title_full_unstemmed | Importance of external validation and subgroup analysis of artificial intelligence in the detection of low ejection fraction from electrocardiograms |
title_short | Importance of external validation and subgroup analysis of artificial intelligence in the detection of low ejection fraction from electrocardiograms |
title_sort | importance of external validation and subgroup analysis of artificial intelligence in the detection of low ejection fraction from electrocardiograms |
topic | Short Report |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9779862/ https://www.ncbi.nlm.nih.gov/pubmed/36710903 http://dx.doi.org/10.1093/ehjdh/ztac065 |
work_keys_str_mv | AT yagiryuichiro importanceofexternalvalidationandsubgroupanalysisofartificialintelligenceinthedetectionoflowejectionfractionfromelectrocardiograms AT gotoshinichi importanceofexternalvalidationandsubgroupanalysisofartificialintelligenceinthedetectionoflowejectionfractionfromelectrocardiograms AT katsumatayoshinori importanceofexternalvalidationandsubgroupanalysisofartificialintelligenceinthedetectionoflowejectionfractionfromelectrocardiograms AT macraecaluma importanceofexternalvalidationandsubgroupanalysisofartificialintelligenceinthedetectionoflowejectionfractionfromelectrocardiograms AT deorahulc importanceofexternalvalidationandsubgroupanalysisofartificialintelligenceinthedetectionoflowejectionfractionfromelectrocardiograms |