Using Clinical Vignettes to Evaluate VTE Protocol Adherence

BACKGROUND: Venous thromboembolism (VTE) prophylaxis is underutilized in hospitalized medical patients. Underutilization might occur as a result of resident practice variation incurred by using a complex risk assessment tool. OBJECTIVE: To examine what impact repetitive exposure to an electronic point-based VTE risk assessment tool has on resident inter-rater reliability and protocol adherence. DESIGN: Pre and post intervention cross-sectional cohort study. SETTING: Single academic center. PATIENTS: Convenience samples of Internal Medicine residents. INTERVENTIONS: Residents completed clinical vignettes before and after any exposure to an electronic risk assessment tool and reminder alert. They were asked to make three determinations using a point-based VTE risk assessment tool: risk stratification, identify contraindications, and VTE prevention strategy. MEASUREMENTS: Inter-rater reliability for risk assessment, contraindications, and VTE prophylaxis strategy and protocol adherence. RESULTS: Kappa scores for VTE risk assessment did not change, but improved for VTE plan increasing from 0.28 to 0.37. Protocol adherence improved from 71% in 2008 to 79% (P = 0.06). There was a significant decrease in under-prophylaxis (22% to 6%, P < 0.0001) but a significant increase in over-prophylaxis (7% to 16%, P = 0.001). CONCLUSIONS: Using clinical vignettes, we determined that daily exposure to an electronic risk assessment tool did not improve the inter-rater reliability of a point-based risk assessment tool when used by medical residents. This might be due to inexperienced providers using a complex point-based tool. Overall, adherence improved, and under-prophylaxis decreased, but over-prophylaxis increased. Clinical vignettes are a generalizable method to monitor resident prophylaxis practices and way to identify educational and process improvement opportunities. KEYWORDS: Resident; Inter-rater reliability; Venous thromboembolism; Agreement; Risk assessment score