Clinically sufficient classification accuracy and key predictors of treatment failure in a randomized controlled trial of Internet-delivered Cognitive Behavior Therapy for Insomnia

BACKGROUND: In Adaptive Treatment Strategies, each patient's outcome is predicted early in treatment, and treatment is adapted for those at risk of failure. It is unclear what minimum accuracy is needed for a classifier to be clinically useful. This study aimed to establish a empirically supported benchmark accuracy for an Adaptive Treatment Strategy and explore the relative value of input predictors. METHOD: Predictions from 200 patients receiving Internet-delivered cognitive-behavioral therapy in an RCT was analyzed. Correlation and logistic regression was used to explore all included predictors and the predictive capacity of different models. RESULTS: The classifier had a Balanced accuracy of 67 %. Eleven out of the 21 predictors correlated significantly with Failure. A model using all predictors explained 56 % of the outcome variance, and simpler models between 16 and 47 %. Important predictors were patient rated stress, treatment credibility, depression change, and insomnia symptoms at week 3 as well as clinician rated attitudes towards homework and sleep medication. CONCLUSIONS: The accuracy (67 %) found in this study sets a minimum benchmark for when prediction accuracy could be clinically useful. Key predictive factors were mainly related to insomnia, depression or treatment involvement. Simpler predictive models showed some promise and should be developed further, possibly using machine learning methods.