Cargando…

Knowledge distillation with ensembles of convolutional neural networks for medical image segmentation

PURPOSE: Ensembles of convolutional neural networks (CNNs) often outperform a single CNN in medical image segmentation tasks, but inference is computationally more expensive and makes ensembles unattractive for some applications. We compared the performance of differently constructed ensembles with...

Descripción completa

Detalles Bibliográficos
Autores principales: Noothout, Julia M. H., Lessmann, Nikolas, van Eede, Matthijs C., van Harten, Louis D., Sogancioglu, Ecem, Heslinga, Friso G., Veta, Mitko, van Ginneken, Bram, Išgum, Ivana
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Society of Photo-Optical Instrumentation Engineers 2022
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9142841/
https://www.ncbi.nlm.nih.gov/pubmed/35692896
http://dx.doi.org/10.1117/1.JMI.9.5.052407
_version_ 1784715659861557248
author Noothout, Julia M. H.
Lessmann, Nikolas
van Eede, Matthijs C.
van Harten, Louis D.
Sogancioglu, Ecem
Heslinga, Friso G.
Veta, Mitko
van Ginneken, Bram
Išgum, Ivana
author_facet Noothout, Julia M. H.
Lessmann, Nikolas
van Eede, Matthijs C.
van Harten, Louis D.
Sogancioglu, Ecem
Heslinga, Friso G.
Veta, Mitko
van Ginneken, Bram
Išgum, Ivana
author_sort Noothout, Julia M. H.
collection PubMed
description PURPOSE: Ensembles of convolutional neural networks (CNNs) often outperform a single CNN in medical image segmentation tasks, but inference is computationally more expensive and makes ensembles unattractive for some applications. We compared the performance of differently constructed ensembles with the performance of CNNs derived from these ensembles using knowledge distillation, a technique for reducing the footprint of large models such as ensembles. APPROACH: We investigated two different types of ensembles, namely, diverse ensembles of networks with three different architectures and two different loss-functions, and uniform ensembles of networks with the same architecture but initialized with different random seeds. For each ensemble, additionally, a single student network was trained to mimic the class probabilities predicted by the teacher model, the ensemble. We evaluated the performance of each network, the ensembles, and the corresponding distilled networks across three different publicly available datasets. These included chest computed tomography scans with four annotated organs of interest, brain magnetic resonance imaging (MRI) with six annotated brain structures, and cardiac cine-MRI with three annotated heart structures. RESULTS: Both uniform and diverse ensembles obtained better results than any of the individual networks in the ensemble. Furthermore, applying knowledge distillation resulted in a single network that was smaller and faster without compromising performance compared with the ensemble it learned from. The distilled networks significantly outperformed the same network trained with reference segmentation instead of knowledge distillation. CONCLUSION: Knowledge distillation can compress segmentation ensembles of uniform or diverse composition into a single CNN while maintaining the performance of the ensemble.
format Online
Article
Text
id pubmed-9142841
institution National Center for Biotechnology Information
language English
publishDate 2022
publisher Society of Photo-Optical Instrumentation Engineers
record_format MEDLINE/PubMed
spelling pubmed-91428412023-05-28 Knowledge distillation with ensembles of convolutional neural networks for medical image segmentation Noothout, Julia M. H. Lessmann, Nikolas van Eede, Matthijs C. van Harten, Louis D. Sogancioglu, Ecem Heslinga, Friso G. Veta, Mitko van Ginneken, Bram Išgum, Ivana J Med Imaging (Bellingham) Special Section on Advances in High Dimensional Medical Imaging PURPOSE: Ensembles of convolutional neural networks (CNNs) often outperform a single CNN in medical image segmentation tasks, but inference is computationally more expensive and makes ensembles unattractive for some applications. We compared the performance of differently constructed ensembles with the performance of CNNs derived from these ensembles using knowledge distillation, a technique for reducing the footprint of large models such as ensembles. APPROACH: We investigated two different types of ensembles, namely, diverse ensembles of networks with three different architectures and two different loss-functions, and uniform ensembles of networks with the same architecture but initialized with different random seeds. For each ensemble, additionally, a single student network was trained to mimic the class probabilities predicted by the teacher model, the ensemble. We evaluated the performance of each network, the ensembles, and the corresponding distilled networks across three different publicly available datasets. These included chest computed tomography scans with four annotated organs of interest, brain magnetic resonance imaging (MRI) with six annotated brain structures, and cardiac cine-MRI with three annotated heart structures. RESULTS: Both uniform and diverse ensembles obtained better results than any of the individual networks in the ensemble. Furthermore, applying knowledge distillation resulted in a single network that was smaller and faster without compromising performance compared with the ensemble it learned from. The distilled networks significantly outperformed the same network trained with reference segmentation instead of knowledge distillation. CONCLUSION: Knowledge distillation can compress segmentation ensembles of uniform or diverse composition into a single CNN while maintaining the performance of the ensemble. Society of Photo-Optical Instrumentation Engineers 2022-05-28 2022-09 /pmc/articles/PMC9142841/ /pubmed/35692896 http://dx.doi.org/10.1117/1.JMI.9.5.052407 Text en © 2022 The Authors https://creativecommons.org/licenses/by/4.0/Published by SPIE under a Creative Commons Attribution 4.0 International License. Distribution or reproduction of this work in whole or in part requires full attribution of the original publication, including its DOI.
spellingShingle Special Section on Advances in High Dimensional Medical Imaging
Noothout, Julia M. H.
Lessmann, Nikolas
van Eede, Matthijs C.
van Harten, Louis D.
Sogancioglu, Ecem
Heslinga, Friso G.
Veta, Mitko
van Ginneken, Bram
Išgum, Ivana
Knowledge distillation with ensembles of convolutional neural networks for medical image segmentation
title Knowledge distillation with ensembles of convolutional neural networks for medical image segmentation
title_full Knowledge distillation with ensembles of convolutional neural networks for medical image segmentation
title_fullStr Knowledge distillation with ensembles of convolutional neural networks for medical image segmentation
title_full_unstemmed Knowledge distillation with ensembles of convolutional neural networks for medical image segmentation
title_short Knowledge distillation with ensembles of convolutional neural networks for medical image segmentation
title_sort knowledge distillation with ensembles of convolutional neural networks for medical image segmentation
topic Special Section on Advances in High Dimensional Medical Imaging
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9142841/
https://www.ncbi.nlm.nih.gov/pubmed/35692896
http://dx.doi.org/10.1117/1.JMI.9.5.052407
work_keys_str_mv AT noothoutjuliamh knowledgedistillationwithensemblesofconvolutionalneuralnetworksformedicalimagesegmentation
AT lessmannnikolas knowledgedistillationwithensemblesofconvolutionalneuralnetworksformedicalimagesegmentation
AT vaneedematthijsc knowledgedistillationwithensemblesofconvolutionalneuralnetworksformedicalimagesegmentation
AT vanhartenlouisd knowledgedistillationwithensemblesofconvolutionalneuralnetworksformedicalimagesegmentation
AT soganciogluecem knowledgedistillationwithensemblesofconvolutionalneuralnetworksformedicalimagesegmentation
AT heslingafrisog knowledgedistillationwithensemblesofconvolutionalneuralnetworksformedicalimagesegmentation
AT vetamitko knowledgedistillationwithensemblesofconvolutionalneuralnetworksformedicalimagesegmentation
AT vanginnekenbram knowledgedistillationwithensemblesofconvolutionalneuralnetworksformedicalimagesegmentation
AT isgumivana knowledgedistillationwithensemblesofconvolutionalneuralnetworksformedicalimagesegmentation