Cargando…

A Trust-Based Methodology to Evaluate Deep Learning Models for Automatic Diagnosis of Ocular Toxoplasmosis from Fundus Images

In the automatic diagnosis of ocular toxoplasmosis (OT), Deep Learning (DL) has arisen as a powerful and promising approach for diagnosis. However, despite the good performance of the models, decision rules should be interpretable to elicit trust from the medical community. Therefore, the developmen...

Descripción completa

Detalles Bibliográficos
Autores principales: Parra, Rodrigo, Ojeda, Verena, Vázquez Noguera, Jose Luis, García-Torres, Miguel, Mello-Román, Julio César, Villalba, Cynthia, Facon, Jacques, Divina, Federico, Cardozo, Olivia, Castillo, Verónica Elisa, Matto, Ingrid Castro
Formato: Online Artículo Texto
Lenguaje:English
Publicado: MDPI 2021
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8617957/
https://www.ncbi.nlm.nih.gov/pubmed/34829299
http://dx.doi.org/10.3390/diagnostics11111951
_version_ 1784604631842684928
author Parra, Rodrigo
Ojeda, Verena
Vázquez Noguera, Jose Luis
García-Torres, Miguel
Mello-Román, Julio César
Villalba, Cynthia
Facon, Jacques
Divina, Federico
Cardozo, Olivia
Castillo, Verónica Elisa
Matto, Ingrid Castro
author_facet Parra, Rodrigo
Ojeda, Verena
Vázquez Noguera, Jose Luis
García-Torres, Miguel
Mello-Román, Julio César
Villalba, Cynthia
Facon, Jacques
Divina, Federico
Cardozo, Olivia
Castillo, Verónica Elisa
Matto, Ingrid Castro
author_sort Parra, Rodrigo
collection PubMed
description In the automatic diagnosis of ocular toxoplasmosis (OT), Deep Learning (DL) has arisen as a powerful and promising approach for diagnosis. However, despite the good performance of the models, decision rules should be interpretable to elicit trust from the medical community. Therefore, the development of an evaluation methodology to assess DL models based on interpretability methods is a challenging task that is necessary to extend the use of AI among clinicians. In this work, we propose a novel methodology to quantify the similarity between the decision rules used by a DL model and an ophthalmologist, based on the assumption that doctors are more likely to trust a prediction that was based on decision rules they can understand. Given an eye fundus image with OT, the proposed methodology compares the segmentation mask of OT lesions labeled by an ophthalmologist with the attribution matrix produced by interpretability methods. Furthermore, an open dataset that includes the eye fundus images and the segmentation masks is shared with the community. The proposal was tested on three different DL architectures. The results suggest that complex models tend to perform worse in terms of likelihood to be trusted while achieving better results in sensitivity and specificity.
format Online
Article
Text
id pubmed-8617957
institution National Center for Biotechnology Information
language English
publishDate 2021
publisher MDPI
record_format MEDLINE/PubMed
spelling pubmed-86179572021-11-27 A Trust-Based Methodology to Evaluate Deep Learning Models for Automatic Diagnosis of Ocular Toxoplasmosis from Fundus Images Parra, Rodrigo Ojeda, Verena Vázquez Noguera, Jose Luis García-Torres, Miguel Mello-Román, Julio César Villalba, Cynthia Facon, Jacques Divina, Federico Cardozo, Olivia Castillo, Verónica Elisa Matto, Ingrid Castro Diagnostics (Basel) Article In the automatic diagnosis of ocular toxoplasmosis (OT), Deep Learning (DL) has arisen as a powerful and promising approach for diagnosis. However, despite the good performance of the models, decision rules should be interpretable to elicit trust from the medical community. Therefore, the development of an evaluation methodology to assess DL models based on interpretability methods is a challenging task that is necessary to extend the use of AI among clinicians. In this work, we propose a novel methodology to quantify the similarity between the decision rules used by a DL model and an ophthalmologist, based on the assumption that doctors are more likely to trust a prediction that was based on decision rules they can understand. Given an eye fundus image with OT, the proposed methodology compares the segmentation mask of OT lesions labeled by an ophthalmologist with the attribution matrix produced by interpretability methods. Furthermore, an open dataset that includes the eye fundus images and the segmentation masks is shared with the community. The proposal was tested on three different DL architectures. The results suggest that complex models tend to perform worse in terms of likelihood to be trusted while achieving better results in sensitivity and specificity. MDPI 2021-10-21 /pmc/articles/PMC8617957/ /pubmed/34829299 http://dx.doi.org/10.3390/diagnostics11111951 Text en © 2021 by the authors. https://creativecommons.org/licenses/by/4.0/Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
spellingShingle Article
Parra, Rodrigo
Ojeda, Verena
Vázquez Noguera, Jose Luis
García-Torres, Miguel
Mello-Román, Julio César
Villalba, Cynthia
Facon, Jacques
Divina, Federico
Cardozo, Olivia
Castillo, Verónica Elisa
Matto, Ingrid Castro
A Trust-Based Methodology to Evaluate Deep Learning Models for Automatic Diagnosis of Ocular Toxoplasmosis from Fundus Images
title A Trust-Based Methodology to Evaluate Deep Learning Models for Automatic Diagnosis of Ocular Toxoplasmosis from Fundus Images
title_full A Trust-Based Methodology to Evaluate Deep Learning Models for Automatic Diagnosis of Ocular Toxoplasmosis from Fundus Images
title_fullStr A Trust-Based Methodology to Evaluate Deep Learning Models for Automatic Diagnosis of Ocular Toxoplasmosis from Fundus Images
title_full_unstemmed A Trust-Based Methodology to Evaluate Deep Learning Models for Automatic Diagnosis of Ocular Toxoplasmosis from Fundus Images
title_short A Trust-Based Methodology to Evaluate Deep Learning Models for Automatic Diagnosis of Ocular Toxoplasmosis from Fundus Images
title_sort trust-based methodology to evaluate deep learning models for automatic diagnosis of ocular toxoplasmosis from fundus images
topic Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8617957/
https://www.ncbi.nlm.nih.gov/pubmed/34829299
http://dx.doi.org/10.3390/diagnostics11111951
work_keys_str_mv AT parrarodrigo atrustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages
AT ojedaverena atrustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages
AT vazqueznoguerajoseluis atrustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages
AT garciatorresmiguel atrustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages
AT melloromanjuliocesar atrustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages
AT villalbacynthia atrustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages
AT faconjacques atrustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages
AT divinafederico atrustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages
AT cardozoolivia atrustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages
AT castilloveronicaelisa atrustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages
AT mattoingridcastro atrustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages
AT parrarodrigo trustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages
AT ojedaverena trustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages
AT vazqueznoguerajoseluis trustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages
AT garciatorresmiguel trustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages
AT melloromanjuliocesar trustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages
AT villalbacynthia trustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages
AT faconjacques trustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages
AT divinafederico trustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages
AT cardozoolivia trustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages
AT castilloveronicaelisa trustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages
AT mattoingridcastro trustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages