Cargando…

Empirical evidence of the impact of study characteristics on the performance of prediction models: a meta-epidemiological study

OBJECTIVES: To empirically assess the relation between study characteristics and prognostic model performance in external validation studies of multivariable prognostic models. DESIGN: Meta-epidemiological study. DATA SOURCES AND STUDY SELECTION: On 16 October 2018, we searched electronic databases...

Descripción completa

Detalles Bibliográficos
Autores principales: Damen, Johanna A A G, Debray, Thomas P A, Pajouheshnia, Romin, Reitsma, Johannes B, Scholten, Rob J P M, Moons, Karel G M, Hooft, Lotty
Formato: Online Artículo Texto
Lenguaje:English
Publicado: BMJ Publishing Group 2019
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6500242/
https://www.ncbi.nlm.nih.gov/pubmed/30940759
http://dx.doi.org/10.1136/bmjopen-2018-026160
_version_ 1783415910839615488
author Damen, Johanna A A G
Debray, Thomas P A
Pajouheshnia, Romin
Reitsma, Johannes B
Scholten, Rob J P M
Moons, Karel G M
Hooft, Lotty
author_facet Damen, Johanna A A G
Debray, Thomas P A
Pajouheshnia, Romin
Reitsma, Johannes B
Scholten, Rob J P M
Moons, Karel G M
Hooft, Lotty
author_sort Damen, Johanna A A G
collection PubMed
description OBJECTIVES: To empirically assess the relation between study characteristics and prognostic model performance in external validation studies of multivariable prognostic models. DESIGN: Meta-epidemiological study. DATA SOURCES AND STUDY SELECTION: On 16 October 2018, we searched electronic databases for systematic reviews of prognostic models. Reviews from non-overlapping clinical fields were selected if they reported common performance measures (either the concordance (c)-statistic or the ratio of observed over expected number of events (OE ratio)) from 10 or more validations of the same prognostic model. DATA EXTRACTION AND ANALYSES: Study design features, population characteristics, methods of predictor and outcome assessment, and the aforementioned performance measures were extracted from the included external validation studies. Random effects meta-regression was used to quantify the association between the study characteristics and model performance. RESULTS: We included 10 systematic reviews, describing a total of 224 external validations, of which 221 reported c-statistics and 124 OE ratios. Associations between study characteristics and model performance were heterogeneous across systematic reviews. C-statistics were most associated with variation in population characteristics, outcome definitions and measurement and predictor substitution. For example, validations with eligibility criteria comparable to the development study were associated with higher c-statistics compared with narrower criteria (difference in logit c-statistic 0.21(95% CI 0.07 to 0.35), similar to an increase from 0.70 to 0.74). Using a case-control design was associated with higher OE ratios, compared with using data from a cohort (difference in log OE ratio 0.97(95% CI 0.38 to 1.55), similar to an increase in OE ratio from 1.00 to 2.63). CONCLUSIONS: Variation in performance of prognostic models across studies is mainly associated with variation in case-mix, study designs, outcome definitions and measurement methods and predictor substitution. Researchers developing and validating prognostic models should realise the potential influence of these study characteristics on the predictive performance of prognostic models.
format Online
Article
Text
id pubmed-6500242
institution National Center for Biotechnology Information
language English
publishDate 2019
publisher BMJ Publishing Group
record_format MEDLINE/PubMed
spelling pubmed-65002422019-05-21 Empirical evidence of the impact of study characteristics on the performance of prediction models: a meta-epidemiological study Damen, Johanna A A G Debray, Thomas P A Pajouheshnia, Romin Reitsma, Johannes B Scholten, Rob J P M Moons, Karel G M Hooft, Lotty BMJ Open Research Methods OBJECTIVES: To empirically assess the relation between study characteristics and prognostic model performance in external validation studies of multivariable prognostic models. DESIGN: Meta-epidemiological study. DATA SOURCES AND STUDY SELECTION: On 16 October 2018, we searched electronic databases for systematic reviews of prognostic models. Reviews from non-overlapping clinical fields were selected if they reported common performance measures (either the concordance (c)-statistic or the ratio of observed over expected number of events (OE ratio)) from 10 or more validations of the same prognostic model. DATA EXTRACTION AND ANALYSES: Study design features, population characteristics, methods of predictor and outcome assessment, and the aforementioned performance measures were extracted from the included external validation studies. Random effects meta-regression was used to quantify the association between the study characteristics and model performance. RESULTS: We included 10 systematic reviews, describing a total of 224 external validations, of which 221 reported c-statistics and 124 OE ratios. Associations between study characteristics and model performance were heterogeneous across systematic reviews. C-statistics were most associated with variation in population characteristics, outcome definitions and measurement and predictor substitution. For example, validations with eligibility criteria comparable to the development study were associated with higher c-statistics compared with narrower criteria (difference in logit c-statistic 0.21(95% CI 0.07 to 0.35), similar to an increase from 0.70 to 0.74). Using a case-control design was associated with higher OE ratios, compared with using data from a cohort (difference in log OE ratio 0.97(95% CI 0.38 to 1.55), similar to an increase in OE ratio from 1.00 to 2.63). CONCLUSIONS: Variation in performance of prognostic models across studies is mainly associated with variation in case-mix, study designs, outcome definitions and measurement methods and predictor substitution. Researchers developing and validating prognostic models should realise the potential influence of these study characteristics on the predictive performance of prognostic models. BMJ Publishing Group 2019-04-01 /pmc/articles/PMC6500242/ /pubmed/30940759 http://dx.doi.org/10.1136/bmjopen-2018-026160 Text en © Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ. This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
spellingShingle Research Methods
Damen, Johanna A A G
Debray, Thomas P A
Pajouheshnia, Romin
Reitsma, Johannes B
Scholten, Rob J P M
Moons, Karel G M
Hooft, Lotty
Empirical evidence of the impact of study characteristics on the performance of prediction models: a meta-epidemiological study
title Empirical evidence of the impact of study characteristics on the performance of prediction models: a meta-epidemiological study
title_full Empirical evidence of the impact of study characteristics on the performance of prediction models: a meta-epidemiological study
title_fullStr Empirical evidence of the impact of study characteristics on the performance of prediction models: a meta-epidemiological study
title_full_unstemmed Empirical evidence of the impact of study characteristics on the performance of prediction models: a meta-epidemiological study
title_short Empirical evidence of the impact of study characteristics on the performance of prediction models: a meta-epidemiological study
title_sort empirical evidence of the impact of study characteristics on the performance of prediction models: a meta-epidemiological study
topic Research Methods
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6500242/
https://www.ncbi.nlm.nih.gov/pubmed/30940759
http://dx.doi.org/10.1136/bmjopen-2018-026160
work_keys_str_mv AT damenjohannaaag empiricalevidenceoftheimpactofstudycharacteristicsontheperformanceofpredictionmodelsametaepidemiologicalstudy
AT debraythomaspa empiricalevidenceoftheimpactofstudycharacteristicsontheperformanceofpredictionmodelsametaepidemiologicalstudy
AT pajouheshniaromin empiricalevidenceoftheimpactofstudycharacteristicsontheperformanceofpredictionmodelsametaepidemiologicalstudy
AT reitsmajohannesb empiricalevidenceoftheimpactofstudycharacteristicsontheperformanceofpredictionmodelsametaepidemiologicalstudy
AT scholtenrobjpm empiricalevidenceoftheimpactofstudycharacteristicsontheperformanceofpredictionmodelsametaepidemiologicalstudy
AT moonskarelgm empiricalevidenceoftheimpactofstudycharacteristicsontheperformanceofpredictionmodelsametaepidemiologicalstudy
AT hooftlotty empiricalevidenceoftheimpactofstudycharacteristicsontheperformanceofpredictionmodelsametaepidemiologicalstudy