MAEPI and CIR: New Metrics for Robust Evaluation of the Prediction Performance of AI-Based IOL Formulas

PURPOSE: To develop a class of new metrics for evaluating the performance of intraocular lens power calculation formulas robust to issues that can arise with AI-based methods. METHODS: The dataset consists of surgical information and biometry measurements of 6893 eyes of 5016 cataract patients who received Alcon SN60WF lenses at University of Michigan's Kellogg Eye Center. We designed two types of new metrics: the MAEPI (Mean Absolute Error in Prediction of Intraocular Lens [IOL]) and the CIR (Correct IOL Rate) and compared the new metrics with traditional metrics including the mean absolute error (MAE), median absolute error, and standard deviation. We evaluated the new metrics with simulation analysis, machine learning (ML) methods, as well as existing IOL formulas (Barrett Universal II, Haigis, Hoffer Q, Holladay 1, PearlDGS, and SRK/T). RESULTS: Results of traditional metrics did not accurately reflect the performance of overfitted ML formulas. By contrast, MAEPI and CIR discriminated between accurate and inaccurate formulas. The standard IOL formulas received low MAEPI and high CIR, which were consistent with the results of the traditional metrics. CONCLUSIONS: MAEPI and CIR provide a more accurate reflection of the real-life performance of AI-based IOL formula than traditional metrics. They should be computed in conjunction with conventional metrics when evaluating the performance of new and existing IOL formulas. TRANSLATIONAL RELEVANCE: The proposed new metrics would help cataract patients avoid the risks caused by inaccurate AI-based formulas, whose true performance cannot be determined by traditional metrics.