Cargando…
A Trust-Based Methodology to Evaluate Deep Learning Models for Automatic Diagnosis of Ocular Toxoplasmosis from Fundus Images
In the automatic diagnosis of ocular toxoplasmosis (OT), Deep Learning (DL) has arisen as a powerful and promising approach for diagnosis. However, despite the good performance of the models, decision rules should be interpretable to elicit trust from the medical community. Therefore, the developmen...
Autores principales: | , , , , , , , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
MDPI
2021
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8617957/ https://www.ncbi.nlm.nih.gov/pubmed/34829299 http://dx.doi.org/10.3390/diagnostics11111951 |
_version_ | 1784604631842684928 |
---|---|
author | Parra, Rodrigo Ojeda, Verena Vázquez Noguera, Jose Luis García-Torres, Miguel Mello-Román, Julio César Villalba, Cynthia Facon, Jacques Divina, Federico Cardozo, Olivia Castillo, Verónica Elisa Matto, Ingrid Castro |
author_facet | Parra, Rodrigo Ojeda, Verena Vázquez Noguera, Jose Luis García-Torres, Miguel Mello-Román, Julio César Villalba, Cynthia Facon, Jacques Divina, Federico Cardozo, Olivia Castillo, Verónica Elisa Matto, Ingrid Castro |
author_sort | Parra, Rodrigo |
collection | PubMed |
description | In the automatic diagnosis of ocular toxoplasmosis (OT), Deep Learning (DL) has arisen as a powerful and promising approach for diagnosis. However, despite the good performance of the models, decision rules should be interpretable to elicit trust from the medical community. Therefore, the development of an evaluation methodology to assess DL models based on interpretability methods is a challenging task that is necessary to extend the use of AI among clinicians. In this work, we propose a novel methodology to quantify the similarity between the decision rules used by a DL model and an ophthalmologist, based on the assumption that doctors are more likely to trust a prediction that was based on decision rules they can understand. Given an eye fundus image with OT, the proposed methodology compares the segmentation mask of OT lesions labeled by an ophthalmologist with the attribution matrix produced by interpretability methods. Furthermore, an open dataset that includes the eye fundus images and the segmentation masks is shared with the community. The proposal was tested on three different DL architectures. The results suggest that complex models tend to perform worse in terms of likelihood to be trusted while achieving better results in sensitivity and specificity. |
format | Online Article Text |
id | pubmed-8617957 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2021 |
publisher | MDPI |
record_format | MEDLINE/PubMed |
spelling | pubmed-86179572021-11-27 A Trust-Based Methodology to Evaluate Deep Learning Models for Automatic Diagnosis of Ocular Toxoplasmosis from Fundus Images Parra, Rodrigo Ojeda, Verena Vázquez Noguera, Jose Luis García-Torres, Miguel Mello-Román, Julio César Villalba, Cynthia Facon, Jacques Divina, Federico Cardozo, Olivia Castillo, Verónica Elisa Matto, Ingrid Castro Diagnostics (Basel) Article In the automatic diagnosis of ocular toxoplasmosis (OT), Deep Learning (DL) has arisen as a powerful and promising approach for diagnosis. However, despite the good performance of the models, decision rules should be interpretable to elicit trust from the medical community. Therefore, the development of an evaluation methodology to assess DL models based on interpretability methods is a challenging task that is necessary to extend the use of AI among clinicians. In this work, we propose a novel methodology to quantify the similarity between the decision rules used by a DL model and an ophthalmologist, based on the assumption that doctors are more likely to trust a prediction that was based on decision rules they can understand. Given an eye fundus image with OT, the proposed methodology compares the segmentation mask of OT lesions labeled by an ophthalmologist with the attribution matrix produced by interpretability methods. Furthermore, an open dataset that includes the eye fundus images and the segmentation masks is shared with the community. The proposal was tested on three different DL architectures. The results suggest that complex models tend to perform worse in terms of likelihood to be trusted while achieving better results in sensitivity and specificity. MDPI 2021-10-21 /pmc/articles/PMC8617957/ /pubmed/34829299 http://dx.doi.org/10.3390/diagnostics11111951 Text en © 2021 by the authors. https://creativecommons.org/licenses/by/4.0/Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/). |
spellingShingle | Article Parra, Rodrigo Ojeda, Verena Vázquez Noguera, Jose Luis García-Torres, Miguel Mello-Román, Julio César Villalba, Cynthia Facon, Jacques Divina, Federico Cardozo, Olivia Castillo, Verónica Elisa Matto, Ingrid Castro A Trust-Based Methodology to Evaluate Deep Learning Models for Automatic Diagnosis of Ocular Toxoplasmosis from Fundus Images |
title | A Trust-Based Methodology to Evaluate Deep Learning Models for Automatic Diagnosis of Ocular Toxoplasmosis from Fundus Images |
title_full | A Trust-Based Methodology to Evaluate Deep Learning Models for Automatic Diagnosis of Ocular Toxoplasmosis from Fundus Images |
title_fullStr | A Trust-Based Methodology to Evaluate Deep Learning Models for Automatic Diagnosis of Ocular Toxoplasmosis from Fundus Images |
title_full_unstemmed | A Trust-Based Methodology to Evaluate Deep Learning Models for Automatic Diagnosis of Ocular Toxoplasmosis from Fundus Images |
title_short | A Trust-Based Methodology to Evaluate Deep Learning Models for Automatic Diagnosis of Ocular Toxoplasmosis from Fundus Images |
title_sort | trust-based methodology to evaluate deep learning models for automatic diagnosis of ocular toxoplasmosis from fundus images |
topic | Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8617957/ https://www.ncbi.nlm.nih.gov/pubmed/34829299 http://dx.doi.org/10.3390/diagnostics11111951 |
work_keys_str_mv | AT parrarodrigo atrustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages AT ojedaverena atrustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages AT vazqueznoguerajoseluis atrustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages AT garciatorresmiguel atrustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages AT melloromanjuliocesar atrustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages AT villalbacynthia atrustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages AT faconjacques atrustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages AT divinafederico atrustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages AT cardozoolivia atrustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages AT castilloveronicaelisa atrustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages AT mattoingridcastro atrustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages AT parrarodrigo trustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages AT ojedaverena trustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages AT vazqueznoguerajoseluis trustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages AT garciatorresmiguel trustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages AT melloromanjuliocesar trustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages AT villalbacynthia trustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages AT faconjacques trustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages AT divinafederico trustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages AT cardozoolivia trustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages AT castilloveronicaelisa trustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages AT mattoingridcastro trustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages |