Cargando…
Expert validation of prediction models for a clinical decision-support system in audiology
For supporting clinical decision-making in audiology, Common Audiological Functional Parameters (CAFPAs) were suggested as an interpretable intermediate representation of audiological information taken from various diagnostic sources within a clinical decision-support system (CDSS). Ten different CA...
Autores principales: | , , , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Frontiers Media S.A.
2022
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9446152/ https://www.ncbi.nlm.nih.gov/pubmed/36081868 http://dx.doi.org/10.3389/fneur.2022.960012 |
_version_ | 1784783585592475648 |
---|---|
author | Buhl, Mareike Akin, Gülce Saak, Samira Eysholdt, Ulrich Radeloff, Andreas Kollmeier, Birger Hildebrandt, Andrea |
author_facet | Buhl, Mareike Akin, Gülce Saak, Samira Eysholdt, Ulrich Radeloff, Andreas Kollmeier, Birger Hildebrandt, Andrea |
author_sort | Buhl, Mareike |
collection | PubMed |
description | For supporting clinical decision-making in audiology, Common Audiological Functional Parameters (CAFPAs) were suggested as an interpretable intermediate representation of audiological information taken from various diagnostic sources within a clinical decision-support system (CDSS). Ten different CAFPAs were proposed to represent specific functional aspects of the human auditory system, namely hearing threshold, supra-threshold deficits, binaural hearing, neural processing, cognitive abilities, and a socio-economic component. CAFPAs were established as a viable basis for deriving audiological findings and treatment recommendations, and it has been demonstrated that model-predicted CAFPAs, with machine learning models trained on expert-labeled patient cases, are sufficiently accurate to be included in a CDSS, but it requires further validation by experts. The present study aimed to validate model-predicted CAFPAs based on previously unlabeled cases from the same data set. Here, we ask to which extent domain experts agree with the model-predicted CAFPAs and whether potential disagreement can be understood in terms of patient characteristics. To these aims, an expert survey was designed and applied to two highly-experienced audiology specialists. They were asked to evaluate model-predicted CAFPAs and estimate audiological findings of the given audiological information about the patients that they were presented with simultaneously. The results revealed strong relative agreement between the two experts and importantly between experts and the prediction for all CAFPAs, except for the neural processing and binaural hearing-related ones. It turned out, however, that experts tend to score CAFPAs in a larger value range, but, on average, across patients with smaller scores as compared with the machine learning models. For the hearing threshold-associated CAFPA in frequencies smaller than 0.75 kHz and the cognitive CAFPA, not only the relative agreement but also the absolute agreement between machine and experts was very high. For those CAFPAs with an average difference between the model- and expert-estimated values, patient characteristics were predictive of the disagreement. The findings are discussed in terms of how they can help toward further improvement of model-predicted CAFPAs to be incorporated in a CDSS for audiology. |
format | Online Article Text |
id | pubmed-9446152 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2022 |
publisher | Frontiers Media S.A. |
record_format | MEDLINE/PubMed |
spelling | pubmed-94461522022-09-07 Expert validation of prediction models for a clinical decision-support system in audiology Buhl, Mareike Akin, Gülce Saak, Samira Eysholdt, Ulrich Radeloff, Andreas Kollmeier, Birger Hildebrandt, Andrea Front Neurol Neurology For supporting clinical decision-making in audiology, Common Audiological Functional Parameters (CAFPAs) were suggested as an interpretable intermediate representation of audiological information taken from various diagnostic sources within a clinical decision-support system (CDSS). Ten different CAFPAs were proposed to represent specific functional aspects of the human auditory system, namely hearing threshold, supra-threshold deficits, binaural hearing, neural processing, cognitive abilities, and a socio-economic component. CAFPAs were established as a viable basis for deriving audiological findings and treatment recommendations, and it has been demonstrated that model-predicted CAFPAs, with machine learning models trained on expert-labeled patient cases, are sufficiently accurate to be included in a CDSS, but it requires further validation by experts. The present study aimed to validate model-predicted CAFPAs based on previously unlabeled cases from the same data set. Here, we ask to which extent domain experts agree with the model-predicted CAFPAs and whether potential disagreement can be understood in terms of patient characteristics. To these aims, an expert survey was designed and applied to two highly-experienced audiology specialists. They were asked to evaluate model-predicted CAFPAs and estimate audiological findings of the given audiological information about the patients that they were presented with simultaneously. The results revealed strong relative agreement between the two experts and importantly between experts and the prediction for all CAFPAs, except for the neural processing and binaural hearing-related ones. It turned out, however, that experts tend to score CAFPAs in a larger value range, but, on average, across patients with smaller scores as compared with the machine learning models. For the hearing threshold-associated CAFPA in frequencies smaller than 0.75 kHz and the cognitive CAFPA, not only the relative agreement but also the absolute agreement between machine and experts was very high. For those CAFPAs with an average difference between the model- and expert-estimated values, patient characteristics were predictive of the disagreement. The findings are discussed in terms of how they can help toward further improvement of model-predicted CAFPAs to be incorporated in a CDSS for audiology. Frontiers Media S.A. 2022-08-23 /pmc/articles/PMC9446152/ /pubmed/36081868 http://dx.doi.org/10.3389/fneur.2022.960012 Text en Copyright © 2022 Buhl, Akin, Saak, Eysholdt, Radeloff, Kollmeier and Hildebrandt. https://creativecommons.org/licenses/by/4.0/This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms. |
spellingShingle | Neurology Buhl, Mareike Akin, Gülce Saak, Samira Eysholdt, Ulrich Radeloff, Andreas Kollmeier, Birger Hildebrandt, Andrea Expert validation of prediction models for a clinical decision-support system in audiology |
title | Expert validation of prediction models for a clinical decision-support system in audiology |
title_full | Expert validation of prediction models for a clinical decision-support system in audiology |
title_fullStr | Expert validation of prediction models for a clinical decision-support system in audiology |
title_full_unstemmed | Expert validation of prediction models for a clinical decision-support system in audiology |
title_short | Expert validation of prediction models for a clinical decision-support system in audiology |
title_sort | expert validation of prediction models for a clinical decision-support system in audiology |
topic | Neurology |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9446152/ https://www.ncbi.nlm.nih.gov/pubmed/36081868 http://dx.doi.org/10.3389/fneur.2022.960012 |
work_keys_str_mv | AT buhlmareike expertvalidationofpredictionmodelsforaclinicaldecisionsupportsysteminaudiology AT akingulce expertvalidationofpredictionmodelsforaclinicaldecisionsupportsysteminaudiology AT saaksamira expertvalidationofpredictionmodelsforaclinicaldecisionsupportsysteminaudiology AT eysholdtulrich expertvalidationofpredictionmodelsforaclinicaldecisionsupportsysteminaudiology AT radeloffandreas expertvalidationofpredictionmodelsforaclinicaldecisionsupportsysteminaudiology AT kollmeierbirger expertvalidationofpredictionmodelsforaclinicaldecisionsupportsysteminaudiology AT hildebrandtandrea expertvalidationofpredictionmodelsforaclinicaldecisionsupportsysteminaudiology |