Cargando…

Discovering Thematically Coherent Biomedical Documents Using Contextualized Bidirectional Encoder Representations from Transformers-Based Clustering

The increasing expansion of biomedical documents has increased the number of natural language textual resources related to the current applications. Meanwhile, there has been a great interest in extracting useful information from meaningful coherent groupings of textual content documents in the last...

Descripción completa

Detalles Bibliográficos
Autores principales: Davagdorj, Khishigsuren, Wang, Ling, Li, Meijing, Pham, Van-Huy, Ryu, Keun Ho, Theera-Umpon, Nipon
Formato: Online Artículo Texto
Lenguaje:English
Publicado: MDPI 2022
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9141535/
https://www.ncbi.nlm.nih.gov/pubmed/35627429
http://dx.doi.org/10.3390/ijerph19105893
_version_ 1784715369179512832
author Davagdorj, Khishigsuren
Wang, Ling
Li, Meijing
Pham, Van-Huy
Ryu, Keun Ho
Theera-Umpon, Nipon
author_facet Davagdorj, Khishigsuren
Wang, Ling
Li, Meijing
Pham, Van-Huy
Ryu, Keun Ho
Theera-Umpon, Nipon
author_sort Davagdorj, Khishigsuren
collection PubMed
description The increasing expansion of biomedical documents has increased the number of natural language textual resources related to the current applications. Meanwhile, there has been a great interest in extracting useful information from meaningful coherent groupings of textual content documents in the last decade. However, it is challenging to discover informative representations and define relevant articles from the rapidly growing biomedical literature due to the unsupervised nature of document clustering. Moreover, empirical investigations demonstrated that traditional text clustering methods produce unsatisfactory results in terms of non-contextualized vector space representations because that neglect the semantic relationship between biomedical texts. Recently, pre-trained language models have emerged as successful in a wide range of natural language processing applications. In this paper, we propose the Gaussian Mixture Model-based efficient clustering framework that incorporates substantially pre-trained (Bidirectional Encoder Representations from Transformers for Biomedical Text Mining) BioBERT domain-specific language representations to enhance the clustering accuracy. Our proposed framework consists of main three phases. First, classic text pre-processing techniques are used biomedical document data, which crawled from the PubMed repository. Second, representative vectors are extracted from a pre-trained BioBERT language model for biomedical text mining. Third, we employ the Gaussian Mixture Model as a clustering algorithm, which allows us to assign labels for each biomedical document. In order to prove the efficiency of our proposed model, we conducted a comprehensive experimental analysis utilizing several clustering algorithms while combining diverse embedding techniques. Consequently, the experimental results show that the proposed model outperforms the benchmark models by reaching performance measures of Fowlkes mallows score, silhouette coefficient, adjusted rand index, Davies-Bouldin score of 0.7817, 0.3765, 0.4478, 1.6849, respectively. We expect the outcomes of this study will assist domain specialists in comprehending thematically cohesive documents in the healthcare field.
format Online
Article
Text
id pubmed-9141535
institution National Center for Biotechnology Information
language English
publishDate 2022
publisher MDPI
record_format MEDLINE/PubMed
spelling pubmed-91415352022-05-28 Discovering Thematically Coherent Biomedical Documents Using Contextualized Bidirectional Encoder Representations from Transformers-Based Clustering Davagdorj, Khishigsuren Wang, Ling Li, Meijing Pham, Van-Huy Ryu, Keun Ho Theera-Umpon, Nipon Int J Environ Res Public Health Article The increasing expansion of biomedical documents has increased the number of natural language textual resources related to the current applications. Meanwhile, there has been a great interest in extracting useful information from meaningful coherent groupings of textual content documents in the last decade. However, it is challenging to discover informative representations and define relevant articles from the rapidly growing biomedical literature due to the unsupervised nature of document clustering. Moreover, empirical investigations demonstrated that traditional text clustering methods produce unsatisfactory results in terms of non-contextualized vector space representations because that neglect the semantic relationship between biomedical texts. Recently, pre-trained language models have emerged as successful in a wide range of natural language processing applications. In this paper, we propose the Gaussian Mixture Model-based efficient clustering framework that incorporates substantially pre-trained (Bidirectional Encoder Representations from Transformers for Biomedical Text Mining) BioBERT domain-specific language representations to enhance the clustering accuracy. Our proposed framework consists of main three phases. First, classic text pre-processing techniques are used biomedical document data, which crawled from the PubMed repository. Second, representative vectors are extracted from a pre-trained BioBERT language model for biomedical text mining. Third, we employ the Gaussian Mixture Model as a clustering algorithm, which allows us to assign labels for each biomedical document. In order to prove the efficiency of our proposed model, we conducted a comprehensive experimental analysis utilizing several clustering algorithms while combining diverse embedding techniques. Consequently, the experimental results show that the proposed model outperforms the benchmark models by reaching performance measures of Fowlkes mallows score, silhouette coefficient, adjusted rand index, Davies-Bouldin score of 0.7817, 0.3765, 0.4478, 1.6849, respectively. We expect the outcomes of this study will assist domain specialists in comprehending thematically cohesive documents in the healthcare field. MDPI 2022-05-12 /pmc/articles/PMC9141535/ /pubmed/35627429 http://dx.doi.org/10.3390/ijerph19105893 Text en © 2022 by the authors. https://creativecommons.org/licenses/by/4.0/Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
spellingShingle Article
Davagdorj, Khishigsuren
Wang, Ling
Li, Meijing
Pham, Van-Huy
Ryu, Keun Ho
Theera-Umpon, Nipon
Discovering Thematically Coherent Biomedical Documents Using Contextualized Bidirectional Encoder Representations from Transformers-Based Clustering
title Discovering Thematically Coherent Biomedical Documents Using Contextualized Bidirectional Encoder Representations from Transformers-Based Clustering
title_full Discovering Thematically Coherent Biomedical Documents Using Contextualized Bidirectional Encoder Representations from Transformers-Based Clustering
title_fullStr Discovering Thematically Coherent Biomedical Documents Using Contextualized Bidirectional Encoder Representations from Transformers-Based Clustering
title_full_unstemmed Discovering Thematically Coherent Biomedical Documents Using Contextualized Bidirectional Encoder Representations from Transformers-Based Clustering
title_short Discovering Thematically Coherent Biomedical Documents Using Contextualized Bidirectional Encoder Representations from Transformers-Based Clustering
title_sort discovering thematically coherent biomedical documents using contextualized bidirectional encoder representations from transformers-based clustering
topic Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9141535/
https://www.ncbi.nlm.nih.gov/pubmed/35627429
http://dx.doi.org/10.3390/ijerph19105893
work_keys_str_mv AT davagdorjkhishigsuren discoveringthematicallycoherentbiomedicaldocumentsusingcontextualizedbidirectionalencoderrepresentationsfromtransformersbasedclustering
AT wangling discoveringthematicallycoherentbiomedicaldocumentsusingcontextualizedbidirectionalencoderrepresentationsfromtransformersbasedclustering
AT limeijing discoveringthematicallycoherentbiomedicaldocumentsusingcontextualizedbidirectionalencoderrepresentationsfromtransformersbasedclustering
AT phamvanhuy discoveringthematicallycoherentbiomedicaldocumentsusingcontextualizedbidirectionalencoderrepresentationsfromtransformersbasedclustering
AT ryukeunho discoveringthematicallycoherentbiomedicaldocumentsusingcontextualizedbidirectionalencoderrepresentationsfromtransformersbasedclustering
AT theeraumponnipon discoveringthematicallycoherentbiomedicaldocumentsusingcontextualizedbidirectionalencoderrepresentationsfromtransformersbasedclustering