Cargando…
The estimation and use of predictions for the assessment of model performance using large samples with multiply imputed data
Multiple imputation can be used as a tool in the process of constructing prediction models in medical and epidemiological studies with missing covariate values. Such models can be used to make predictions for model performance assessment, but the task is made more complicated by the multiple imputat...
Autores principales: | , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
BlackWell Publishing Ltd
2015
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4515100/ https://www.ncbi.nlm.nih.gov/pubmed/25630926 http://dx.doi.org/10.1002/bimj.201400004 |
_version_ | 1782382875050508288 |
---|---|
author | Wood, Angela M Royston, Patrick White, Ian R |
author_facet | Wood, Angela M Royston, Patrick White, Ian R |
author_sort | Wood, Angela M |
collection | PubMed |
description | Multiple imputation can be used as a tool in the process of constructing prediction models in medical and epidemiological studies with missing covariate values. Such models can be used to make predictions for model performance assessment, but the task is made more complicated by the multiple imputation structure. We summarize various predictions constructed from covariates, including multiply imputed covariates, and either the set of imputation-specific prediction model coefficients or the pooled prediction model coefficients. We further describe approaches for using the predictions to assess model performance. We distinguish between ideal model performance and pragmatic model performance, where the former refers to the model's performance in an ideal clinical setting where all individuals have fully observed predictors and the latter refers to the model's performance in a real-world clinical setting where some individuals have missing predictors. The approaches are compared through an extensive simulation study based on the UK700 trial. We determine that measures of ideal model performance can be estimated within imputed datasets and subsequently pooled to give an overall measure of model performance. Alternative methods to evaluate pragmatic model performance are required and we propose constructing predictions either from a second set of covariate imputations which make no use of observed outcomes, or from a set of partial prediction models constructed for each potential observed pattern of covariate. Pragmatic model performance is generally lower than ideal model performance. We focus on model performance within the derivation data, but describe how to extend all the methods to a validation dataset. |
format | Online Article Text |
id | pubmed-4515100 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2015 |
publisher | BlackWell Publishing Ltd |
record_format | MEDLINE/PubMed |
spelling | pubmed-45151002015-07-31 The estimation and use of predictions for the assessment of model performance using large samples with multiply imputed data Wood, Angela M Royston, Patrick White, Ian R Biom J Research Papers Multiple imputation can be used as a tool in the process of constructing prediction models in medical and epidemiological studies with missing covariate values. Such models can be used to make predictions for model performance assessment, but the task is made more complicated by the multiple imputation structure. We summarize various predictions constructed from covariates, including multiply imputed covariates, and either the set of imputation-specific prediction model coefficients or the pooled prediction model coefficients. We further describe approaches for using the predictions to assess model performance. We distinguish between ideal model performance and pragmatic model performance, where the former refers to the model's performance in an ideal clinical setting where all individuals have fully observed predictors and the latter refers to the model's performance in a real-world clinical setting where some individuals have missing predictors. The approaches are compared through an extensive simulation study based on the UK700 trial. We determine that measures of ideal model performance can be estimated within imputed datasets and subsequently pooled to give an overall measure of model performance. Alternative methods to evaluate pragmatic model performance are required and we propose constructing predictions either from a second set of covariate imputations which make no use of observed outcomes, or from a set of partial prediction models constructed for each potential observed pattern of covariate. Pragmatic model performance is generally lower than ideal model performance. We focus on model performance within the derivation data, but describe how to extend all the methods to a validation dataset. BlackWell Publishing Ltd 2015-07 2015-01-29 /pmc/articles/PMC4515100/ /pubmed/25630926 http://dx.doi.org/10.1002/bimj.201400004 Text en © 2015 The Author. Biometrical Journal published by WILEY-VCH Verlag GmbH & Co. KGaA, Weinheim http://creativecommons.org/licenses/by/4.0/ This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited. |
spellingShingle | Research Papers Wood, Angela M Royston, Patrick White, Ian R The estimation and use of predictions for the assessment of model performance using large samples with multiply imputed data |
title | The estimation and use of predictions for the assessment of model performance using large samples with multiply imputed data |
title_full | The estimation and use of predictions for the assessment of model performance using large samples with multiply imputed data |
title_fullStr | The estimation and use of predictions for the assessment of model performance using large samples with multiply imputed data |
title_full_unstemmed | The estimation and use of predictions for the assessment of model performance using large samples with multiply imputed data |
title_short | The estimation and use of predictions for the assessment of model performance using large samples with multiply imputed data |
title_sort | estimation and use of predictions for the assessment of model performance using large samples with multiply imputed data |
topic | Research Papers |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4515100/ https://www.ncbi.nlm.nih.gov/pubmed/25630926 http://dx.doi.org/10.1002/bimj.201400004 |
work_keys_str_mv | AT woodangelam theestimationanduseofpredictionsfortheassessmentofmodelperformanceusinglargesampleswithmultiplyimputeddata AT roystonpatrick theestimationanduseofpredictionsfortheassessmentofmodelperformanceusinglargesampleswithmultiplyimputeddata AT whiteianr theestimationanduseofpredictionsfortheassessmentofmodelperformanceusinglargesampleswithmultiplyimputeddata AT woodangelam estimationanduseofpredictionsfortheassessmentofmodelperformanceusinglargesampleswithmultiplyimputeddata AT roystonpatrick estimationanduseofpredictionsfortheassessmentofmodelperformanceusinglargesampleswithmultiplyimputeddata AT whiteianr estimationanduseofpredictionsfortheassessmentofmodelperformanceusinglargesampleswithmultiplyimputeddata |