Assessment of the Utility of Social Media for Broad-Ranging Statistical Signal Detection in Pharmacovigilance: Results from the WEB-RADR Project
INTRODUCTION AND OBJECTIVE: Social media has been proposed as a possibly useful data source for pharmacovigilance signal detection. This study primarily aimed to evaluate the performance of established statistical signal detection algorithms in Twitter/Facebook for a broad range of drugs and adverse...
Autores principales: | , , , , , , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Springer International Publishing
2018
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6223695/ https://www.ncbi.nlm.nih.gov/pubmed/30043385 http://dx.doi.org/10.1007/s40264-018-0699-2 |
_version_ | 1783369446690127872 |
---|---|
author | Caster, Ola Dietrich, Juergen Kürzinger, Marie-Laure Lerch, Magnus Maskell, Simon Norén, G. Niklas Tcherny-Lessenot, Stéphanie Vroman, Benoit Wisniewski, Antoni van Stekelenborg, John |
author_facet | Caster, Ola Dietrich, Juergen Kürzinger, Marie-Laure Lerch, Magnus Maskell, Simon Norén, G. Niklas Tcherny-Lessenot, Stéphanie Vroman, Benoit Wisniewski, Antoni van Stekelenborg, John |
author_sort | Caster, Ola |
collection | PubMed |
description | INTRODUCTION AND OBJECTIVE: Social media has been proposed as a possibly useful data source for pharmacovigilance signal detection. This study primarily aimed to evaluate the performance of established statistical signal detection algorithms in Twitter/Facebook for a broad range of drugs and adverse events. METHODS: Performance was assessed using a reference set by Harpaz et al., consisting of 62 US Food and Drug Administration labelling changes, and an internal WEB-RADR reference set consisting of 200 validated safety signals. In total, 75 drugs were studied. Twitter/Facebook posts were retrieved for the period March 2012 to March 2015, and drugs/events were extracted from the posts. We retrieved 4.3 million and 2.0 million posts for the WEB-RADR and Harpaz drugs, respectively. Individual case reports were extracted from VigiBase for the same period. Disproportionality algorithms based on the Information Component or the Proportional Reporting Ratio and crude post/report counting were applied in Twitter/Facebook and VigiBase. Receiver operating characteristic curves were generated, and the relative timing of alerting was analysed. RESULTS: Across all algorithms, the area under the receiver operating characteristic curve for Twitter/Facebook varied between 0.47 and 0.53 for the WEB-RADR reference set and between 0.48 and 0.53 for the Harpaz reference set. For VigiBase, the ranges were 0.64–0.69 and 0.55–0.67, respectively. In Twitter/Facebook, at best, 31 (16%) and four (6%) positive controls were detected prior to their index dates in the WEB-RADR and Harpaz references, respectively. In VigiBase, the corresponding numbers were 66 (33%) and 17 (27%). CONCLUSIONS: Our results clearly suggest that broad-ranging statistical signal detection in Twitter and Facebook, using currently available methods for adverse event recognition, performs poorly and cannot be recommended at the expense of other pharmacovigilance activities. ELECTRONIC SUPPLEMENTARY MATERIAL: The online version of this article (10.1007/s40264-018-0699-2) contains supplementary material, which is available to authorized users. |
format | Online Article Text |
id | pubmed-6223695 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2018 |
publisher | Springer International Publishing |
record_format | MEDLINE/PubMed |
spelling | pubmed-62236952018-11-18 Assessment of the Utility of Social Media for Broad-Ranging Statistical Signal Detection in Pharmacovigilance: Results from the WEB-RADR Project Caster, Ola Dietrich, Juergen Kürzinger, Marie-Laure Lerch, Magnus Maskell, Simon Norén, G. Niklas Tcherny-Lessenot, Stéphanie Vroman, Benoit Wisniewski, Antoni van Stekelenborg, John Drug Saf Original Research Article INTRODUCTION AND OBJECTIVE: Social media has been proposed as a possibly useful data source for pharmacovigilance signal detection. This study primarily aimed to evaluate the performance of established statistical signal detection algorithms in Twitter/Facebook for a broad range of drugs and adverse events. METHODS: Performance was assessed using a reference set by Harpaz et al., consisting of 62 US Food and Drug Administration labelling changes, and an internal WEB-RADR reference set consisting of 200 validated safety signals. In total, 75 drugs were studied. Twitter/Facebook posts were retrieved for the period March 2012 to March 2015, and drugs/events were extracted from the posts. We retrieved 4.3 million and 2.0 million posts for the WEB-RADR and Harpaz drugs, respectively. Individual case reports were extracted from VigiBase for the same period. Disproportionality algorithms based on the Information Component or the Proportional Reporting Ratio and crude post/report counting were applied in Twitter/Facebook and VigiBase. Receiver operating characteristic curves were generated, and the relative timing of alerting was analysed. RESULTS: Across all algorithms, the area under the receiver operating characteristic curve for Twitter/Facebook varied between 0.47 and 0.53 for the WEB-RADR reference set and between 0.48 and 0.53 for the Harpaz reference set. For VigiBase, the ranges were 0.64–0.69 and 0.55–0.67, respectively. In Twitter/Facebook, at best, 31 (16%) and four (6%) positive controls were detected prior to their index dates in the WEB-RADR and Harpaz references, respectively. In VigiBase, the corresponding numbers were 66 (33%) and 17 (27%). CONCLUSIONS: Our results clearly suggest that broad-ranging statistical signal detection in Twitter and Facebook, using currently available methods for adverse event recognition, performs poorly and cannot be recommended at the expense of other pharmacovigilance activities. ELECTRONIC SUPPLEMENTARY MATERIAL: The online version of this article (10.1007/s40264-018-0699-2) contains supplementary material, which is available to authorized users. Springer International Publishing 2018-07-24 2018 /pmc/articles/PMC6223695/ /pubmed/30043385 http://dx.doi.org/10.1007/s40264-018-0699-2 Text en © The Author(s) 2018 Open AccessThis article is distributed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncommercial use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. |
spellingShingle | Original Research Article Caster, Ola Dietrich, Juergen Kürzinger, Marie-Laure Lerch, Magnus Maskell, Simon Norén, G. Niklas Tcherny-Lessenot, Stéphanie Vroman, Benoit Wisniewski, Antoni van Stekelenborg, John Assessment of the Utility of Social Media for Broad-Ranging Statistical Signal Detection in Pharmacovigilance: Results from the WEB-RADR Project |
title | Assessment of the Utility of Social Media for Broad-Ranging Statistical Signal Detection in Pharmacovigilance: Results from the WEB-RADR Project |
title_full | Assessment of the Utility of Social Media for Broad-Ranging Statistical Signal Detection in Pharmacovigilance: Results from the WEB-RADR Project |
title_fullStr | Assessment of the Utility of Social Media for Broad-Ranging Statistical Signal Detection in Pharmacovigilance: Results from the WEB-RADR Project |
title_full_unstemmed | Assessment of the Utility of Social Media for Broad-Ranging Statistical Signal Detection in Pharmacovigilance: Results from the WEB-RADR Project |
title_short | Assessment of the Utility of Social Media for Broad-Ranging Statistical Signal Detection in Pharmacovigilance: Results from the WEB-RADR Project |
title_sort | assessment of the utility of social media for broad-ranging statistical signal detection in pharmacovigilance: results from the web-radr project |
topic | Original Research Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6223695/ https://www.ncbi.nlm.nih.gov/pubmed/30043385 http://dx.doi.org/10.1007/s40264-018-0699-2 |
work_keys_str_mv | AT casterola assessmentoftheutilityofsocialmediaforbroadrangingstatisticalsignaldetectioninpharmacovigilanceresultsfromthewebradrproject AT dietrichjuergen assessmentoftheutilityofsocialmediaforbroadrangingstatisticalsignaldetectioninpharmacovigilanceresultsfromthewebradrproject AT kurzingermarielaure assessmentoftheutilityofsocialmediaforbroadrangingstatisticalsignaldetectioninpharmacovigilanceresultsfromthewebradrproject AT lerchmagnus assessmentoftheutilityofsocialmediaforbroadrangingstatisticalsignaldetectioninpharmacovigilanceresultsfromthewebradrproject AT maskellsimon assessmentoftheutilityofsocialmediaforbroadrangingstatisticalsignaldetectioninpharmacovigilanceresultsfromthewebradrproject AT norengniklas assessmentoftheutilityofsocialmediaforbroadrangingstatisticalsignaldetectioninpharmacovigilanceresultsfromthewebradrproject AT tchernylessenotstephanie assessmentoftheutilityofsocialmediaforbroadrangingstatisticalsignaldetectioninpharmacovigilanceresultsfromthewebradrproject AT vromanbenoit assessmentoftheutilityofsocialmediaforbroadrangingstatisticalsignaldetectioninpharmacovigilanceresultsfromthewebradrproject AT wisniewskiantoni assessmentoftheutilityofsocialmediaforbroadrangingstatisticalsignaldetectioninpharmacovigilanceresultsfromthewebradrproject AT vanstekelenborgjohn assessmentoftheutilityofsocialmediaforbroadrangingstatisticalsignaldetectioninpharmacovigilanceresultsfromthewebradrproject |