Automated assessment of the smoothness of retinal layers in optical coherence tomography images using a machine learning algorithm

Quantifying the smoothness of different layers of the retina can potentially be an important and practical biomarker in various pathologic conditions like diabetic retinopathy. The purpose of this study is to develop an automated machine learning algorithm which uses support vector regression method with wavelet kernel and automatically segments two hyperreflective retinal layers (inner plexiform layer (IPL) and outer plexiform layer (OPL)) in 50 optical coherence tomography (OCT) slabs and calculates the smoothness index (SI). The Bland–Altman plots, mean absolute error, root mean square error and signed error calculations revealed a modest discrepancy between the manual approach, used as the ground truth, and the corresponding automated segmentation of IPL/ OPL, as well as SI measurements in OCT slabs. It was concluded that the constructed algorithm may be employed as a reliable, rapid and convenient approach for segmenting IPL/OPL and calculating SI in the appropriate layers. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s12880-023-00976-w.