Cargando…

Mitigating Biases in CORD-19 for Analyzing COVID-19 Literature

On the behest of the Office of Science and Technology Policy in the White House, six institutions, including ours, have created an open research dataset called COVID-19 Research Dataset (CORD-19) to facilitate the development of question-answering systems that can assist researchers in finding relev...

Descripción completa

Detalles Bibliográficos
Autores principales: Kanakia, Anshul, Wang, Kuansan, Dong, Yuxiao, Xie, Boya, Lo, Kyle, Shen, Zhihong, Wang, Lucy Lu, Huang, Chiyuan, Eide, Darrin, Kohlmeier, Sebastian, Wu, Chieh-Han
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Frontiers Media S.A. 2020
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8025972/
https://www.ncbi.nlm.nih.gov/pubmed/33870059
http://dx.doi.org/10.3389/frma.2020.596624
_version_ 1783675589468618752
author Kanakia, Anshul
Wang, Kuansan
Dong, Yuxiao
Xie, Boya
Lo, Kyle
Shen, Zhihong
Wang, Lucy Lu
Huang, Chiyuan
Eide, Darrin
Kohlmeier, Sebastian
Wu, Chieh-Han
author_facet Kanakia, Anshul
Wang, Kuansan
Dong, Yuxiao
Xie, Boya
Lo, Kyle
Shen, Zhihong
Wang, Lucy Lu
Huang, Chiyuan
Eide, Darrin
Kohlmeier, Sebastian
Wu, Chieh-Han
author_sort Kanakia, Anshul
collection PubMed
description On the behest of the Office of Science and Technology Policy in the White House, six institutions, including ours, have created an open research dataset called COVID-19 Research Dataset (CORD-19) to facilitate the development of question-answering systems that can assist researchers in finding relevant research on COVID-19. As of May 27, 2020, CORD-19 includes more than 100,000 open access publications from major publishers and PubMed as well as preprint articles deposited into medRxiv, bioRxiv, and arXiv. Recent years, however, have also seen question-answering and other machine learning systems exhibit harmful behaviors to humans due to biases in the training data. It is imperative and only ethical for modern scientists to be vigilant in inspecting and be prepared to mitigate the potential biases when working with any datasets. This article describes a framework to examine biases in scientific document collections like CORD-19 by comparing their properties with those derived from the citation behaviors of the entire scientific community. In total, three expanded sets are created for the analyses: 1) the enclosure set CORD-19E composed of CORD-19 articles and their references and citations, mirroring the methodology used in the renowned “A Century of Physics” analysis; 2) the full closure graph CORD-19C that recursively includes references starting with CORD-19; and 3) the inflection closure CORD-19I, that is, a much smaller subset of CORD-19C but already appropriate for statistical analysis based on the theory of the scale-free nature of the citation network. Taken together, all these expanded datasets show much smoother trends when used to analyze global COVID-19 research. The results suggest that while CORD-19 exhibits a strong tilt toward recent and topically focused articles, the knowledge being explored to attack the pandemic encompasses a much longer time span and is very interdisciplinary. A question-answering system with such expanded scope of knowledge may perform better in understanding the literature and answering related questions. However, while CORD-19 appears to have topical coverage biases compared to the expanded sets, the collaboration patterns, especially in terms of team sizes and geographical distributions, are captured very well already in CORD-19 as the raw statistics and trends agree with those from larger datasets.
format Online
Article
Text
id pubmed-8025972
institution National Center for Biotechnology Information
language English
publishDate 2020
publisher Frontiers Media S.A.
record_format MEDLINE/PubMed
spelling pubmed-80259722021-04-15 Mitigating Biases in CORD-19 for Analyzing COVID-19 Literature Kanakia, Anshul Wang, Kuansan Dong, Yuxiao Xie, Boya Lo, Kyle Shen, Zhihong Wang, Lucy Lu Huang, Chiyuan Eide, Darrin Kohlmeier, Sebastian Wu, Chieh-Han Front Res Metr Anal Research Metrics and Analytics On the behest of the Office of Science and Technology Policy in the White House, six institutions, including ours, have created an open research dataset called COVID-19 Research Dataset (CORD-19) to facilitate the development of question-answering systems that can assist researchers in finding relevant research on COVID-19. As of May 27, 2020, CORD-19 includes more than 100,000 open access publications from major publishers and PubMed as well as preprint articles deposited into medRxiv, bioRxiv, and arXiv. Recent years, however, have also seen question-answering and other machine learning systems exhibit harmful behaviors to humans due to biases in the training data. It is imperative and only ethical for modern scientists to be vigilant in inspecting and be prepared to mitigate the potential biases when working with any datasets. This article describes a framework to examine biases in scientific document collections like CORD-19 by comparing their properties with those derived from the citation behaviors of the entire scientific community. In total, three expanded sets are created for the analyses: 1) the enclosure set CORD-19E composed of CORD-19 articles and their references and citations, mirroring the methodology used in the renowned “A Century of Physics” analysis; 2) the full closure graph CORD-19C that recursively includes references starting with CORD-19; and 3) the inflection closure CORD-19I, that is, a much smaller subset of CORD-19C but already appropriate for statistical analysis based on the theory of the scale-free nature of the citation network. Taken together, all these expanded datasets show much smoother trends when used to analyze global COVID-19 research. The results suggest that while CORD-19 exhibits a strong tilt toward recent and topically focused articles, the knowledge being explored to attack the pandemic encompasses a much longer time span and is very interdisciplinary. A question-answering system with such expanded scope of knowledge may perform better in understanding the literature and answering related questions. However, while CORD-19 appears to have topical coverage biases compared to the expanded sets, the collaboration patterns, especially in terms of team sizes and geographical distributions, are captured very well already in CORD-19 as the raw statistics and trends agree with those from larger datasets. Frontiers Media S.A. 2020-11-23 /pmc/articles/PMC8025972/ /pubmed/33870059 http://dx.doi.org/10.3389/frma.2020.596624 Text en Copyright © 2020 Kanakia, Wang, Dong, Xie, Lo, Shen, Wang, Huang, Eide, Kohlmeier and Wu https://creativecommons.org/licenses/by/4.0/This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) (https://creativecommons.org/licenses/by/4.0/) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
spellingShingle Research Metrics and Analytics
Kanakia, Anshul
Wang, Kuansan
Dong, Yuxiao
Xie, Boya
Lo, Kyle
Shen, Zhihong
Wang, Lucy Lu
Huang, Chiyuan
Eide, Darrin
Kohlmeier, Sebastian
Wu, Chieh-Han
Mitigating Biases in CORD-19 for Analyzing COVID-19 Literature
title Mitigating Biases in CORD-19 for Analyzing COVID-19 Literature
title_full Mitigating Biases in CORD-19 for Analyzing COVID-19 Literature
title_fullStr Mitigating Biases in CORD-19 for Analyzing COVID-19 Literature
title_full_unstemmed Mitigating Biases in CORD-19 for Analyzing COVID-19 Literature
title_short Mitigating Biases in CORD-19 for Analyzing COVID-19 Literature
title_sort mitigating biases in cord-19 for analyzing covid-19 literature
topic Research Metrics and Analytics
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8025972/
https://www.ncbi.nlm.nih.gov/pubmed/33870059
http://dx.doi.org/10.3389/frma.2020.596624
work_keys_str_mv AT kanakiaanshul mitigatingbiasesincord19foranalyzingcovid19literature
AT wangkuansan mitigatingbiasesincord19foranalyzingcovid19literature
AT dongyuxiao mitigatingbiasesincord19foranalyzingcovid19literature
AT xieboya mitigatingbiasesincord19foranalyzingcovid19literature
AT lokyle mitigatingbiasesincord19foranalyzingcovid19literature
AT shenzhihong mitigatingbiasesincord19foranalyzingcovid19literature
AT wanglucylu mitigatingbiasesincord19foranalyzingcovid19literature
AT huangchiyuan mitigatingbiasesincord19foranalyzingcovid19literature
AT eidedarrin mitigatingbiasesincord19foranalyzingcovid19literature
AT kohlmeiersebastian mitigatingbiasesincord19foranalyzingcovid19literature
AT wuchiehhan mitigatingbiasesincord19foranalyzingcovid19literature