Cargando…

Categorizing Vaccine Confidence With a Transformer-Based Machine Learning Model: Analysis of Nuances of Vaccine Sentiment in Twitter Discourse

BACKGROUND: Social media has become an established platform for individuals to discuss and debate various subjects, including vaccination. With growing conversations on the web and less than desired maternal vaccination uptake rates, these conversations could provide useful insights to inform future...

Descripción completa

Detalles Bibliográficos
Autores principales: Kummervold, Per E, Martin, Sam, Dada, Sara, Kilich, Eliz, Denny, Chermain, Paterson, Pauline, Larson, Heidi J
Formato: Online Artículo Texto
Lenguaje:English
Publicado: JMIR Publications 2021
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8538052/
https://www.ncbi.nlm.nih.gov/pubmed/34623312
http://dx.doi.org/10.2196/29584
_version_ 1784588415703973888
author Kummervold, Per E
Martin, Sam
Dada, Sara
Kilich, Eliz
Denny, Chermain
Paterson, Pauline
Larson, Heidi J
author_facet Kummervold, Per E
Martin, Sam
Dada, Sara
Kilich, Eliz
Denny, Chermain
Paterson, Pauline
Larson, Heidi J
author_sort Kummervold, Per E
collection PubMed
description BACKGROUND: Social media has become an established platform for individuals to discuss and debate various subjects, including vaccination. With growing conversations on the web and less than desired maternal vaccination uptake rates, these conversations could provide useful insights to inform future interventions. However, owing to the volume of web-based posts, manual annotation and analysis are difficult and time consuming. Automated processes for this type of analysis, such as natural language processing, have faced challenges in extracting complex stances such as attitudes toward vaccination from large amounts of text. OBJECTIVE: The aim of this study is to build upon recent advances in transposer-based machine learning methods and test whether transformer-based machine learning could be used as a tool to assess the stance expressed in social media posts toward vaccination during pregnancy. METHODS: A total of 16,604 tweets posted between November 1, 2018, and April 30, 2019, were selected using keyword searches related to maternal vaccination. After excluding irrelevant tweets, the remaining tweets were coded by 3 individual researchers into the categories Promotional, Discouraging, Ambiguous, and Neutral or No Stance. After creating a final data set of 2722 unique tweets, multiple machine learning techniques were trained on a part of this data set and then tested and compared with the human annotators. RESULTS: We found the accuracy of the machine learning techniques to be 81.8% (F score=0.78) compared with the agreed score among the 3 annotators. For comparison, the accuracies of the individual annotators compared with the final score were 83.3%, 77.9%, and 77.5%. CONCLUSIONS: This study demonstrates that we are able to achieve close to the same accuracy in categorizing tweets using our machine learning models as could be expected from a single human coder. The potential to use this automated process, which is reliable and accurate, could free valuable time and resources for conducting this analysis, in addition to informing potentially effective and necessary interventions.
format Online
Article
Text
id pubmed-8538052
institution National Center for Biotechnology Information
language English
publishDate 2021
publisher JMIR Publications
record_format MEDLINE/PubMed
spelling pubmed-85380522021-11-09 Categorizing Vaccine Confidence With a Transformer-Based Machine Learning Model: Analysis of Nuances of Vaccine Sentiment in Twitter Discourse Kummervold, Per E Martin, Sam Dada, Sara Kilich, Eliz Denny, Chermain Paterson, Pauline Larson, Heidi J JMIR Med Inform Original Paper BACKGROUND: Social media has become an established platform for individuals to discuss and debate various subjects, including vaccination. With growing conversations on the web and less than desired maternal vaccination uptake rates, these conversations could provide useful insights to inform future interventions. However, owing to the volume of web-based posts, manual annotation and analysis are difficult and time consuming. Automated processes for this type of analysis, such as natural language processing, have faced challenges in extracting complex stances such as attitudes toward vaccination from large amounts of text. OBJECTIVE: The aim of this study is to build upon recent advances in transposer-based machine learning methods and test whether transformer-based machine learning could be used as a tool to assess the stance expressed in social media posts toward vaccination during pregnancy. METHODS: A total of 16,604 tweets posted between November 1, 2018, and April 30, 2019, were selected using keyword searches related to maternal vaccination. After excluding irrelevant tweets, the remaining tweets were coded by 3 individual researchers into the categories Promotional, Discouraging, Ambiguous, and Neutral or No Stance. After creating a final data set of 2722 unique tweets, multiple machine learning techniques were trained on a part of this data set and then tested and compared with the human annotators. RESULTS: We found the accuracy of the machine learning techniques to be 81.8% (F score=0.78) compared with the agreed score among the 3 annotators. For comparison, the accuracies of the individual annotators compared with the final score were 83.3%, 77.9%, and 77.5%. CONCLUSIONS: This study demonstrates that we are able to achieve close to the same accuracy in categorizing tweets using our machine learning models as could be expected from a single human coder. The potential to use this automated process, which is reliable and accurate, could free valuable time and resources for conducting this analysis, in addition to informing potentially effective and necessary interventions. JMIR Publications 2021-10-08 /pmc/articles/PMC8538052/ /pubmed/34623312 http://dx.doi.org/10.2196/29584 Text en ©Per E Kummervold, Sam Martin, Sara Dada, Eliz Kilich, Chermain Denny, Pauline Paterson, Heidi J Larson. Originally published in JMIR Medical Informatics (https://medinform.jmir.org), 08.10.2021. https://creativecommons.org/licenses/by/4.0/This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on https://medinform.jmir.org/, as well as this copyright and license information must be included.
spellingShingle Original Paper
Kummervold, Per E
Martin, Sam
Dada, Sara
Kilich, Eliz
Denny, Chermain
Paterson, Pauline
Larson, Heidi J
Categorizing Vaccine Confidence With a Transformer-Based Machine Learning Model: Analysis of Nuances of Vaccine Sentiment in Twitter Discourse
title Categorizing Vaccine Confidence With a Transformer-Based Machine Learning Model: Analysis of Nuances of Vaccine Sentiment in Twitter Discourse
title_full Categorizing Vaccine Confidence With a Transformer-Based Machine Learning Model: Analysis of Nuances of Vaccine Sentiment in Twitter Discourse
title_fullStr Categorizing Vaccine Confidence With a Transformer-Based Machine Learning Model: Analysis of Nuances of Vaccine Sentiment in Twitter Discourse
title_full_unstemmed Categorizing Vaccine Confidence With a Transformer-Based Machine Learning Model: Analysis of Nuances of Vaccine Sentiment in Twitter Discourse
title_short Categorizing Vaccine Confidence With a Transformer-Based Machine Learning Model: Analysis of Nuances of Vaccine Sentiment in Twitter Discourse
title_sort categorizing vaccine confidence with a transformer-based machine learning model: analysis of nuances of vaccine sentiment in twitter discourse
topic Original Paper
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8538052/
https://www.ncbi.nlm.nih.gov/pubmed/34623312
http://dx.doi.org/10.2196/29584
work_keys_str_mv AT kummervoldpere categorizingvaccineconfidencewithatransformerbasedmachinelearningmodelanalysisofnuancesofvaccinesentimentintwitterdiscourse
AT martinsam categorizingvaccineconfidencewithatransformerbasedmachinelearningmodelanalysisofnuancesofvaccinesentimentintwitterdiscourse
AT dadasara categorizingvaccineconfidencewithatransformerbasedmachinelearningmodelanalysisofnuancesofvaccinesentimentintwitterdiscourse
AT kilicheliz categorizingvaccineconfidencewithatransformerbasedmachinelearningmodelanalysisofnuancesofvaccinesentimentintwitterdiscourse
AT dennychermain categorizingvaccineconfidencewithatransformerbasedmachinelearningmodelanalysisofnuancesofvaccinesentimentintwitterdiscourse
AT patersonpauline categorizingvaccineconfidencewithatransformerbasedmachinelearningmodelanalysisofnuancesofvaccinesentimentintwitterdiscourse
AT larsonheidij categorizingvaccineconfidencewithatransformerbasedmachinelearningmodelanalysisofnuancesofvaccinesentimentintwitterdiscourse