Assessment of the Utility of Social Media for Broad-Ranging Statistical Signal Detection in Pharmacovigilance: Results from the WEB-RADR Project

INTRODUCTION AND OBJECTIVE: Social media has been proposed as a possibly useful data source for pharmacovigilance signal detection. This study primarily aimed to evaluate the performance of established statistical signal detection algorithms in Twitter/Facebook for a broad range of drugs and adverse...

Descripción completa

Detalles Bibliográficos
Autores principales: Caster, Ola, Dietrich, Juergen, Kürzinger, Marie-Laure, Lerch, Magnus, Maskell, Simon, Norén, G. Niklas, Tcherny-Lessenot, Stéphanie, Vroman, Benoit, Wisniewski, Antoni, van Stekelenborg, John
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Springer International Publishing 2018
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6223695/
https://www.ncbi.nlm.nih.gov/pubmed/30043385
http://dx.doi.org/10.1007/s40264-018-0699-2
_version_ 1783369446690127872
author Caster, Ola
Dietrich, Juergen
Kürzinger, Marie-Laure
Lerch, Magnus
Maskell, Simon
Norén, G. Niklas
Tcherny-Lessenot, Stéphanie
Vroman, Benoit
Wisniewski, Antoni
van Stekelenborg, John
author_facet Caster, Ola
Dietrich, Juergen
Kürzinger, Marie-Laure
Lerch, Magnus
Maskell, Simon
Norén, G. Niklas
Tcherny-Lessenot, Stéphanie
Vroman, Benoit
Wisniewski, Antoni
van Stekelenborg, John
author_sort Caster, Ola
collection PubMed
description INTRODUCTION AND OBJECTIVE: Social media has been proposed as a possibly useful data source for pharmacovigilance signal detection. This study primarily aimed to evaluate the performance of established statistical signal detection algorithms in Twitter/Facebook for a broad range of drugs and adverse events. METHODS: Performance was assessed using a reference set by Harpaz et al., consisting of 62 US Food and Drug Administration labelling changes, and an internal WEB-RADR reference set consisting of 200 validated safety signals. In total, 75 drugs were studied. Twitter/Facebook posts were retrieved for the period March 2012 to March 2015, and drugs/events were extracted from the posts. We retrieved 4.3 million and 2.0 million posts for the WEB-RADR and Harpaz drugs, respectively. Individual case reports were extracted from VigiBase for the same period. Disproportionality algorithms based on the Information Component or the Proportional Reporting Ratio and crude post/report counting were applied in Twitter/Facebook and VigiBase. Receiver operating characteristic curves were generated, and the relative timing of alerting was analysed. RESULTS: Across all algorithms, the area under the receiver operating characteristic curve for Twitter/Facebook varied between 0.47 and 0.53 for the WEB-RADR reference set and between 0.48 and 0.53 for the Harpaz reference set. For VigiBase, the ranges were 0.64–0.69 and 0.55–0.67, respectively. In Twitter/Facebook, at best, 31 (16%) and four (6%) positive controls were detected prior to their index dates in the WEB-RADR and Harpaz references, respectively. In VigiBase, the corresponding numbers were 66 (33%) and 17 (27%). CONCLUSIONS: Our results clearly suggest that broad-ranging statistical signal detection in Twitter and Facebook, using currently available methods for adverse event recognition, performs poorly and cannot be recommended at the expense of other pharmacovigilance activities. ELECTRONIC SUPPLEMENTARY MATERIAL: The online version of this article (10.1007/s40264-018-0699-2) contains supplementary material, which is available to authorized users.
format Online
Article
Text
id pubmed-6223695
institution National Center for Biotechnology Information
language English
publishDate 2018
publisher Springer International Publishing
record_format MEDLINE/PubMed
spelling pubmed-62236952018-11-18 Assessment of the Utility of Social Media for Broad-Ranging Statistical Signal Detection in Pharmacovigilance: Results from the WEB-RADR Project Caster, Ola Dietrich, Juergen Kürzinger, Marie-Laure Lerch, Magnus Maskell, Simon Norén, G. Niklas Tcherny-Lessenot, Stéphanie Vroman, Benoit Wisniewski, Antoni van Stekelenborg, John Drug Saf Original Research Article INTRODUCTION AND OBJECTIVE: Social media has been proposed as a possibly useful data source for pharmacovigilance signal detection. This study primarily aimed to evaluate the performance of established statistical signal detection algorithms in Twitter/Facebook for a broad range of drugs and adverse events. METHODS: Performance was assessed using a reference set by Harpaz et al., consisting of 62 US Food and Drug Administration labelling changes, and an internal WEB-RADR reference set consisting of 200 validated safety signals. In total, 75 drugs were studied. Twitter/Facebook posts were retrieved for the period March 2012 to March 2015, and drugs/events were extracted from the posts. We retrieved 4.3 million and 2.0 million posts for the WEB-RADR and Harpaz drugs, respectively. Individual case reports were extracted from VigiBase for the same period. Disproportionality algorithms based on the Information Component or the Proportional Reporting Ratio and crude post/report counting were applied in Twitter/Facebook and VigiBase. Receiver operating characteristic curves were generated, and the relative timing of alerting was analysed. RESULTS: Across all algorithms, the area under the receiver operating characteristic curve for Twitter/Facebook varied between 0.47 and 0.53 for the WEB-RADR reference set and between 0.48 and 0.53 for the Harpaz reference set. For VigiBase, the ranges were 0.64–0.69 and 0.55–0.67, respectively. In Twitter/Facebook, at best, 31 (16%) and four (6%) positive controls were detected prior to their index dates in the WEB-RADR and Harpaz references, respectively. In VigiBase, the corresponding numbers were 66 (33%) and 17 (27%). CONCLUSIONS: Our results clearly suggest that broad-ranging statistical signal detection in Twitter and Facebook, using currently available methods for adverse event recognition, performs poorly and cannot be recommended at the expense of other pharmacovigilance activities. ELECTRONIC SUPPLEMENTARY MATERIAL: The online version of this article (10.1007/s40264-018-0699-2) contains supplementary material, which is available to authorized users. Springer International Publishing 2018-07-24 2018 /pmc/articles/PMC6223695/ /pubmed/30043385 http://dx.doi.org/10.1007/s40264-018-0699-2 Text en © The Author(s) 2018 Open AccessThis article is distributed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncommercial use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
spellingShingle Original Research Article
Caster, Ola
Dietrich, Juergen
Kürzinger, Marie-Laure
Lerch, Magnus
Maskell, Simon
Norén, G. Niklas
Tcherny-Lessenot, Stéphanie
Vroman, Benoit
Wisniewski, Antoni
van Stekelenborg, John
Assessment of the Utility of Social Media for Broad-Ranging Statistical Signal Detection in Pharmacovigilance: Results from the WEB-RADR Project
title Assessment of the Utility of Social Media for Broad-Ranging Statistical Signal Detection in Pharmacovigilance: Results from the WEB-RADR Project
title_full Assessment of the Utility of Social Media for Broad-Ranging Statistical Signal Detection in Pharmacovigilance: Results from the WEB-RADR Project
title_fullStr Assessment of the Utility of Social Media for Broad-Ranging Statistical Signal Detection in Pharmacovigilance: Results from the WEB-RADR Project
title_full_unstemmed Assessment of the Utility of Social Media for Broad-Ranging Statistical Signal Detection in Pharmacovigilance: Results from the WEB-RADR Project
title_short Assessment of the Utility of Social Media for Broad-Ranging Statistical Signal Detection in Pharmacovigilance: Results from the WEB-RADR Project
title_sort assessment of the utility of social media for broad-ranging statistical signal detection in pharmacovigilance: results from the web-radr project
topic Original Research Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6223695/
https://www.ncbi.nlm.nih.gov/pubmed/30043385
http://dx.doi.org/10.1007/s40264-018-0699-2
work_keys_str_mv AT casterola assessmentoftheutilityofsocialmediaforbroadrangingstatisticalsignaldetectioninpharmacovigilanceresultsfromthewebradrproject
AT dietrichjuergen assessmentoftheutilityofsocialmediaforbroadrangingstatisticalsignaldetectioninpharmacovigilanceresultsfromthewebradrproject
AT kurzingermarielaure assessmentoftheutilityofsocialmediaforbroadrangingstatisticalsignaldetectioninpharmacovigilanceresultsfromthewebradrproject
AT lerchmagnus assessmentoftheutilityofsocialmediaforbroadrangingstatisticalsignaldetectioninpharmacovigilanceresultsfromthewebradrproject
AT maskellsimon assessmentoftheutilityofsocialmediaforbroadrangingstatisticalsignaldetectioninpharmacovigilanceresultsfromthewebradrproject
AT norengniklas assessmentoftheutilityofsocialmediaforbroadrangingstatisticalsignaldetectioninpharmacovigilanceresultsfromthewebradrproject
AT tchernylessenotstephanie assessmentoftheutilityofsocialmediaforbroadrangingstatisticalsignaldetectioninpharmacovigilanceresultsfromthewebradrproject
AT vromanbenoit assessmentoftheutilityofsocialmediaforbroadrangingstatisticalsignaldetectioninpharmacovigilanceresultsfromthewebradrproject
AT wisniewskiantoni assessmentoftheutilityofsocialmediaforbroadrangingstatisticalsignaldetectioninpharmacovigilanceresultsfromthewebradrproject
AT vanstekelenborgjohn assessmentoftheutilityofsocialmediaforbroadrangingstatisticalsignaldetectioninpharmacovigilanceresultsfromthewebradrproject