Effective deep learning training for single-image super-resolution in endomicroscopy exploiting video-registration-based reconstruction

PURPOSE: Probe-based confocal laser endomicroscopy (pCLE) is a recent imaging modality that allows performing in vivo optical biopsies. The design of pCLE hardware, and its reliance on an optical fibre bundle, fundamentally limits the image quality with a few tens of thousands fibres, each acting as the equivalent of a single-pixel detector, assembled into a single fibre bundle. Video registration techniques can be used to estimate high-resolution (HR) images by exploiting the temporal information contained in a sequence of low-resolution (LR) images. However, the alignment of LR frames, required for the fusion, is computationally demanding and prone to artefacts. METHODS: In this work, we propose a novel synthetic data generation approach to train exemplar-based Deep Neural Networks (DNNs). HR pCLE images with enhanced quality are recovered by the models trained on pairs of estimated HR images (generated by the video registration algorithm) and realistic synthetic LR images. Performance of three different state-of-the-art DNNs techniques were analysed on a Smart Atlas database of 8806 images from 238 pCLE video sequences. The results were validated through an extensive image quality assessment that takes into account different quality scores, including a Mean Opinion Score (MOS). RESULTS: Results indicate that the proposed solution produces an effective improvement in the quality of the obtained reconstructed image. CONCLUSION: The proposed training strategy and associated DNNs allows us to perform convincing super-resolution of pCLE images. ELECTRONIC SUPPLEMENTARY MATERIAL: The online version of this article (10.1007/s11548-018-1764-0) contains supplementary material, which is available to authorized users.