A novel strategy for evaluating the effects of an electronic test ordering alert message: Optimizing cardiac marker use

BACKGROUND: Laboratory ordering functions within computerized provider order entry (CPOE) systems typically support the display of electronic alert messages to improve test utilization or implement new ordering policies. However, alert strategies have been shown to vary considerably in their success and the characteristics contributing to an alert's success are poorly understood. Improved methodologies are needed to evaluate alerts and their mechanisms of action. MATERIALS AND METHODS: Clinicians order inpatient and emergency department laboratory tests using our institutional CPOE system. We analyzed user interaction data captured by our CPOE system to evaluate how clinicians responded to an alert. We evaluated an alert designed to implement an institutional policy restricting the indications for ordering creatine kinase-MB (CKMB). RESULTS: Within 2 months of alert implementation, CKMB-associated searches declined by 79% with a corresponding decline in CKMB orders. Furthermore, while prior to alert implementation, clinicians searching for CKMB ultimately ordered this test 99% of the time, following implementation, only 60% of CKMB searches ultimately led to CKMB test orders. This difference presumably represents clinicians who reconsidered the need for CKMB in response to the alert, demonstrating the alert's just-in-time advisory capability. In addition, as clinicians repeatedly viewed the alert, there was a “dose-dependant” decrease in the fraction of searches without orders. This presumably reflects the alerting strategy's long-term educational component, as clinicians aware of the new policy will not search for CKMB when not indicated. CONCLUSIONS: Our analytic approach provides insight into the mechanism of a CPOE alert and demonstrates that alerts may act through a combination of just-in-time advice and longer term education. Use of this approach when implementing alerts may prove useful to improve the success of a given alerting strategy.