Clinical application of machine learning and computer vision to indocyanine green quantification for dynamic intraoperative tissue characterisation: how to do it

INTRODUCTION: Indocyanine green (ICG) quantification and assessment by machine learning (ML) could discriminate tissue types through perfusion characterisation, including delineation of malignancy. Here, we detail the important challenges overcome before effective clinical validation of such capability in a prospective patient series of quantitative fluorescence angiograms regarding primary and secondary colorectal neoplasia. METHODS: ICG perfusion videos from 50 patients (37 with benign (13) and malignant (24) rectal tumours and 13 with colorectal liver metastases) of between 2- and 15-min duration following intravenously administered ICG were formally studied (clinicaltrials.gov: NCT04220242). Video quality with respect to interpretative ML reliability was studied observing practical, technical and technological aspects of fluorescence signal acquisition. Investigated parameters included ICG dosing and administration, distance–intensity fluorescent signal variation, tissue and camera movement (including real-time camera tracking) as well as sampling issues with user-selected digital tissue biopsy. Attenuating strategies for the identified problems were developed, applied and evaluated. ML methods to classify extracted data, including datasets with interrupted time-series lengths with inference simulated data were also evaluated. RESULTS: Definable, remediable challenges arose across both rectal and liver cohorts. Varying ICG dose by tissue type was identified as an important feature of real-time fluorescence quantification. Multi-region sampling within a lesion mitigated representation issues whilst distance–intensity relationships, as well as movement-instability issues, were demonstrated and ameliorated with post-processing techniques including normalisation and smoothing of extracted time–fluorescence curves. ML methods (automated feature extraction and classification) enabled ML algorithms glean excellent pathological categorisation results (AUC-ROC > 0.9, 37 rectal lesions) with imputation proving a robust method of compensation for interrupted time-series data with duration discrepancies. CONCLUSION: Purposeful clinical and data-processing protocols enable powerful pathological characterisation with existing clinical systems. Video analysis as shown can inform iterative and definitive clinical validation studies on how to close the translation gap between research applications and real-world, real-time clinical utility. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1007/s00464-023-09963-2.