Measuring improvement in fracture risk prediction for a new risk factor: a simulation

OBJECTIVE: Improvements in clinical risk prediction models for osteoporosis-related fracture can be evaluated using area under the receiver operating characteristic (AUROC) curve and calibration, as well as reclassification statistics such as the net reclassification improvement (NRI) and integrated discrimination improvement (IDI) statistics. Our objective was to compare the performance of these measures for assessing improvements to an existing fracture risk prediction model. We simulated the effect of a new, randomly-generated risk factor on prediction of major osteoporotic fracture (MOF) for the internationally-validated FRAX(®) model in a cohort from the Manitoba Bone Mineral Density (BMD) Registry. RESULTS: The study cohort was comprised of 31,999 women 50+ years of age; 9.9% sustained at least one MOF in a mean follow-up of 8.4 years. The original prediction model had good discriminative performance, with AUROC = 0.706 and calibration (ratio of observed to predicted risk) of 0.990. The addition of the simulated risk factor resulted in improvements in NRI and IDI for most investigated conditions, while AUROC decreased and changes in calibration were negative. Reclassification measures may give different information than discrimination and calibration about the performance of new clinical risk factors.