Cargando…
AutoDiscern: rating the quality of online health information with hierarchical encoder attention-based neural networks
BACKGROUND: Patients increasingly turn to search engines and online content before, or in place of, talking with a health professional. Low quality health information, which is common on the internet, presents risks to the patient in the form of misinformation and a possibly poorer relationship with...
Autores principales: | , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
BioMed Central
2020
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7285491/ https://www.ncbi.nlm.nih.gov/pubmed/32517759 http://dx.doi.org/10.1186/s12911-020-01131-z |
_version_ | 1783544709881266176 |
---|---|
author | Kinkead, Laura Allam, Ahmed Krauthammer, Michael |
author_facet | Kinkead, Laura Allam, Ahmed Krauthammer, Michael |
author_sort | Kinkead, Laura |
collection | PubMed |
description | BACKGROUND: Patients increasingly turn to search engines and online content before, or in place of, talking with a health professional. Low quality health information, which is common on the internet, presents risks to the patient in the form of misinformation and a possibly poorer relationship with their physician. To address this, the DISCERN criteria (developed at University of Oxford) are used to evaluate the quality of online health information. However, patients are unlikely to take the time to apply these criteria to the health websites they visit. METHODS: We built an automated implementation of the DISCERN instrument (Brief version) using machine learning models. We compared the performance of a traditional model (Random Forest) with that of a hierarchical encoder attention-based neural network (HEA) model using two language embeddings, BERT and BioBERT. RESULTS: The HEA BERT and BioBERT models achieved average F1-macro scores across all criteria of 0.75 and 0.74, respectively, outperforming the Random Forest model (average F1-macro = 0.69). Overall, the neural network based models achieved 81% and 86% average accuracy at 100% and 80% coverage, respectively, compared to 94% manual rating accuracy. The attention mechanism implemented in the HEA architectures not only provided ’model explainability’ by identifying reasonable supporting sentences for the documents fulfilling the Brief DISCERN criteria, but also boosted F1 performance by 0.05 compared to the same architecture without an attention mechanism. CONCLUSIONS: Our research suggests that it is feasible to automate online health information quality assessment, which is an important step towards empowering patients to become informed partners in the healthcare process. |
format | Online Article Text |
id | pubmed-7285491 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2020 |
publisher | BioMed Central |
record_format | MEDLINE/PubMed |
spelling | pubmed-72854912020-06-10 AutoDiscern: rating the quality of online health information with hierarchical encoder attention-based neural networks Kinkead, Laura Allam, Ahmed Krauthammer, Michael BMC Med Inform Decis Mak Research Article BACKGROUND: Patients increasingly turn to search engines and online content before, or in place of, talking with a health professional. Low quality health information, which is common on the internet, presents risks to the patient in the form of misinformation and a possibly poorer relationship with their physician. To address this, the DISCERN criteria (developed at University of Oxford) are used to evaluate the quality of online health information. However, patients are unlikely to take the time to apply these criteria to the health websites they visit. METHODS: We built an automated implementation of the DISCERN instrument (Brief version) using machine learning models. We compared the performance of a traditional model (Random Forest) with that of a hierarchical encoder attention-based neural network (HEA) model using two language embeddings, BERT and BioBERT. RESULTS: The HEA BERT and BioBERT models achieved average F1-macro scores across all criteria of 0.75 and 0.74, respectively, outperforming the Random Forest model (average F1-macro = 0.69). Overall, the neural network based models achieved 81% and 86% average accuracy at 100% and 80% coverage, respectively, compared to 94% manual rating accuracy. The attention mechanism implemented in the HEA architectures not only provided ’model explainability’ by identifying reasonable supporting sentences for the documents fulfilling the Brief DISCERN criteria, but also boosted F1 performance by 0.05 compared to the same architecture without an attention mechanism. CONCLUSIONS: Our research suggests that it is feasible to automate online health information quality assessment, which is an important step towards empowering patients to become informed partners in the healthcare process. BioMed Central 2020-06-09 /pmc/articles/PMC7285491/ /pubmed/32517759 http://dx.doi.org/10.1186/s12911-020-01131-z Text en © The Author(s) 2020 Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data. |
spellingShingle | Research Article Kinkead, Laura Allam, Ahmed Krauthammer, Michael AutoDiscern: rating the quality of online health information with hierarchical encoder attention-based neural networks |
title | AutoDiscern: rating the quality of online health information with hierarchical encoder attention-based neural networks |
title_full | AutoDiscern: rating the quality of online health information with hierarchical encoder attention-based neural networks |
title_fullStr | AutoDiscern: rating the quality of online health information with hierarchical encoder attention-based neural networks |
title_full_unstemmed | AutoDiscern: rating the quality of online health information with hierarchical encoder attention-based neural networks |
title_short | AutoDiscern: rating the quality of online health information with hierarchical encoder attention-based neural networks |
title_sort | autodiscern: rating the quality of online health information with hierarchical encoder attention-based neural networks |
topic | Research Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7285491/ https://www.ncbi.nlm.nih.gov/pubmed/32517759 http://dx.doi.org/10.1186/s12911-020-01131-z |
work_keys_str_mv | AT kinkeadlaura autodiscernratingthequalityofonlinehealthinformationwithhierarchicalencoderattentionbasedneuralnetworks AT allamahmed autodiscernratingthequalityofonlinehealthinformationwithhierarchicalencoderattentionbasedneuralnetworks AT krauthammermichael autodiscernratingthequalityofonlinehealthinformationwithhierarchicalencoderattentionbasedneuralnetworks |