Cargando…

A Unified Framework on Generalizability of Clinical Prediction Models

To be useful, clinical prediction models (CPMs) must be generalizable to patients in new settings. Evaluating generalizability of CPMs helps identify spurious relationships in data, provides insights on when they fail, and thus, improves the explainability of the CPMs. There are discontinuities in c...

Descripción completa

Detalles Bibliográficos
Autores principales: Wan, Bohua, Caffo, Brian, Vedula, S. Swaroop
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Frontiers Media S.A. 2022
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9100692/
https://www.ncbi.nlm.nih.gov/pubmed/35573904
http://dx.doi.org/10.3389/frai.2022.872720
_version_ 1784706907645149184
author Wan, Bohua
Caffo, Brian
Vedula, S. Swaroop
author_facet Wan, Bohua
Caffo, Brian
Vedula, S. Swaroop
author_sort Wan, Bohua
collection PubMed
description To be useful, clinical prediction models (CPMs) must be generalizable to patients in new settings. Evaluating generalizability of CPMs helps identify spurious relationships in data, provides insights on when they fail, and thus, improves the explainability of the CPMs. There are discontinuities in concepts related to generalizability of CPMs in the clinical research and machine learning domains. Specifically, conventional statistical reasons to explain poor generalizability such as inadequate model development for the purposes of generalizability, differences in coding of predictors and outcome between development and external datasets, measurement error, inability to measure some predictors, and missing data, all have differing and often complementary treatments, in the two domains. Much of the current machine learning literature on generalizability of CPMs is in terms of dataset shift of which several types have been described. However, little research exists to synthesize concepts in the two domains. Bridging this conceptual discontinuity in the context of CPMs can facilitate systematic development of CPMs and evaluation of their sensitivity to factors that affect generalizability. We survey generalizability and dataset shift in CPMs from both the clinical research and machine learning perspectives, and describe a unifying framework to analyze generalizability of CPMs and to explain their sensitivity to factors affecting it. Our framework leads to a set of signaling statements that can be used to characterize differences between datasets in terms of factors that affect generalizability of the CPMs.
format Online
Article
Text
id pubmed-9100692
institution National Center for Biotechnology Information
language English
publishDate 2022
publisher Frontiers Media S.A.
record_format MEDLINE/PubMed
spelling pubmed-91006922022-05-14 A Unified Framework on Generalizability of Clinical Prediction Models Wan, Bohua Caffo, Brian Vedula, S. Swaroop Front Artif Intell Artificial Intelligence To be useful, clinical prediction models (CPMs) must be generalizable to patients in new settings. Evaluating generalizability of CPMs helps identify spurious relationships in data, provides insights on when they fail, and thus, improves the explainability of the CPMs. There are discontinuities in concepts related to generalizability of CPMs in the clinical research and machine learning domains. Specifically, conventional statistical reasons to explain poor generalizability such as inadequate model development for the purposes of generalizability, differences in coding of predictors and outcome between development and external datasets, measurement error, inability to measure some predictors, and missing data, all have differing and often complementary treatments, in the two domains. Much of the current machine learning literature on generalizability of CPMs is in terms of dataset shift of which several types have been described. However, little research exists to synthesize concepts in the two domains. Bridging this conceptual discontinuity in the context of CPMs can facilitate systematic development of CPMs and evaluation of their sensitivity to factors that affect generalizability. We survey generalizability and dataset shift in CPMs from both the clinical research and machine learning perspectives, and describe a unifying framework to analyze generalizability of CPMs and to explain their sensitivity to factors affecting it. Our framework leads to a set of signaling statements that can be used to characterize differences between datasets in terms of factors that affect generalizability of the CPMs. Frontiers Media S.A. 2022-04-29 /pmc/articles/PMC9100692/ /pubmed/35573904 http://dx.doi.org/10.3389/frai.2022.872720 Text en Copyright © 2022 Wan, Caffo and Vedula. https://creativecommons.org/licenses/by/4.0/This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
spellingShingle Artificial Intelligence
Wan, Bohua
Caffo, Brian
Vedula, S. Swaroop
A Unified Framework on Generalizability of Clinical Prediction Models
title A Unified Framework on Generalizability of Clinical Prediction Models
title_full A Unified Framework on Generalizability of Clinical Prediction Models
title_fullStr A Unified Framework on Generalizability of Clinical Prediction Models
title_full_unstemmed A Unified Framework on Generalizability of Clinical Prediction Models
title_short A Unified Framework on Generalizability of Clinical Prediction Models
title_sort unified framework on generalizability of clinical prediction models
topic Artificial Intelligence
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9100692/
https://www.ncbi.nlm.nih.gov/pubmed/35573904
http://dx.doi.org/10.3389/frai.2022.872720
work_keys_str_mv AT wanbohua aunifiedframeworkongeneralizabilityofclinicalpredictionmodels
AT caffobrian aunifiedframeworkongeneralizabilityofclinicalpredictionmodels
AT vedulasswaroop aunifiedframeworkongeneralizabilityofclinicalpredictionmodels
AT wanbohua unifiedframeworkongeneralizabilityofclinicalpredictionmodels
AT caffobrian unifiedframeworkongeneralizabilityofclinicalpredictionmodels
AT vedulasswaroop unifiedframeworkongeneralizabilityofclinicalpredictionmodels