Measuring case severity: a novel tool for benchmarking and clinical documentation improvement

BACKGROUND: Severity of illness (SOI) is an All Patients Refined Diagnosis Related Groups (APR DRG) modifier based on comorbidity capture. Tracking SOI helps hospitals improve performance and resource distribution. Furthermore, benchmarking SOI plays a key role in Quality Improvement (QI) efforts such as Clinical Documentation Improvement (CDI) programs. The current SOI system highly relies on the 3 M APR DRG grouper that is updated annually, making it difficult to track severity longitudinally and benchmark against hospitals with different patient populations. Here, we describe an alternative SOI scoring system that is grouper-independent and that can be tracked longitudinally. METHODS: Admission data for 2019–2020 U.S. News and World Report Honor Roll facilities were downloaded from the Vizient Clinical Database and split into training and testing datasets. Elixhauser comorbidities, body systems developed from the Healthcare Cost and Utilization Project (HCUP), and ICD-10-CM complication and comorbidity (CC/MCC) indicators were selected as the predictors for orthogonal polynomial regression models to predict patients’ admission and discharge SOI. Receiver operating characteristic (ROC) and Precision-Recall (PR) analysis, and prediction accuracy were used to evaluate model performance. RESULTS: In the training dataset, the full model including both Elixhauser comorbidities and body system CC/MCC indicators had the highest ROC AUC, PR AUC and predication accuracy for both admission (ROC AUC: 92.9%; PR AUC: 91.0%; prediction accuracy: 85.4%) and discharge SOI (ROC AUC: 93.6%; PR AUC: 92.8%; prediction accuracy: 86.2%). The model including only body system CC/MCC indicators had similar performance for admission (ROC AUC: 92.4%; PR AUC: 90.4%; prediction accuracy: 84.8%) and discharge SOI (ROC AUC: 93.1%; PR AUC: 92.2%; prediction accuracy: 85.6%) as the full model. The model including only Elixhauser comorbidities exhibited the lowest performance. Similarly, in the validation dataset, the prediction accuracy was 86.2% for the full model, 85.6% for the body system model, and 79.3% for the comorbidity model. With fewer variables and less model complexity, the body system model was more efficient and was determined to be the optimal model. The probabilities generated from this model, named J_Score and J_Score_POA, successfully measured SOI and had practical applications in assessment of CDI performance. CONCLUSIONS: The J_Scores generated from the body system model have significant value in evaluating admission and discharge severity of illness. We believe that this new scoring system will provide a useful tool for healthcare institutions to benchmark patients’ illness severity and augment Quality Improvement (QI) efforts. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s12913-022-07935-1.