Critical appraisal and external validation of a prognostic model for survival of people living with HIV/AIDS who underwent antiretroviral therapy

BACKGROUND: HIV/AIDS remains a leading cause of death worldwide. Recently, a model has been developed in Wenzhou, China, to predict the survival of people living with HIV/AIDS (PLWHA) who underwent antiretroviral therapy (ART). We aimed to evaluate the methodological quality and validate the model in an external population-based cohort. METHODS: Prediction Model Risk of Bias Assessment Tool (PROBAST) was used to assess the risk of bias of the Wenzhou model. Data were from the National Free Antiretroviral Treatment Program database. We included PLWHA treated between February 2004 and December 2019 in a tertiary hospital in Guangzhou city, China. The endpoint was all-cause deaths and assessed until January 2020. We assessed the discrimination performance of the model by Harrell’s overall C-statistics and time-dependent C-statistics and calibration by comparing observed survival probabilities estimated with the Kaplan–Meier method versus predicted survival probabilities. To assess the potential prediction value of age and gender which were precluded in developing the Wenzhou model, we compared the discriminative ability of the original model with an extended model added with age and gender. RESULTS: Based on PROBAST, the Wenzhou model was rated as high risk of bias in three out of the four domains (selection of participants, definition of outcome, and methods for statistical analysis) mainly because of the misuse of nested case–control design and propensity score matching. In the external validation analysis, 16758 patients were included, among whom 743 patients died (mortality rate 11.41 per 1000 person-years) during follow-up (median 3.41 years, interquartile range 1.64–5.62). The predictor of HIV viral load was missing in 14361 patients (85.7%). The discriminative ability of the Wenzhou model decreased in the external dataset, with the Harrell’s overall C-statistics being 0.76, and time-dependent C-statistics dropping from 0.81 at 6 months to 0.48 at 10 years after ART initiation. The model consistently underestimated the survival, and the level was 6.23%, 10.02%, and 14.82% at 1, 2, and 3 years after ART initiation, respectively. The overall and time-dependent discriminative ability of the model improved after adding age and gender to the original model. CONCLUSION: The Wenzhou prognostic model is at high risk of bias in model development, with inadequate model performance in external validation. Thereby, we could not confirm the validity and extended utility of the Wenzhou model. Future prediction model development and validation studies need to comply with the methodological standards and guidelines specifically developed for prediction models. SUPPLEMENTARY INFORMATION: Supplementary information accompanies this paper at 10.1186/s41512-020-00088-x.