Cargando…
How best to quantify replication success? A simulation study on the comparison of replication success metrics
To overcome the frequently debated crisis of confidence, replicating studies is becoming increasingly more common. Multiple frequentist and Bayesian measures have been proposed to evaluate whether a replication is successful, but little is known about which method best captures replication success....
Autores principales: | , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
The Royal Society
2021
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8131945/ https://www.ncbi.nlm.nih.gov/pubmed/34017596 http://dx.doi.org/10.1098/rsos.201697 |
_version_ | 1783694816847069184 |
---|---|
author | Muradchanian, Jasmine Hoekstra, Rink Kiers, Henk van Ravenzwaaij, Don |
author_facet | Muradchanian, Jasmine Hoekstra, Rink Kiers, Henk van Ravenzwaaij, Don |
author_sort | Muradchanian, Jasmine |
collection | PubMed |
description | To overcome the frequently debated crisis of confidence, replicating studies is becoming increasingly more common. Multiple frequentist and Bayesian measures have been proposed to evaluate whether a replication is successful, but little is known about which method best captures replication success. This study is one of the first attempts to compare a number of quantitative measures of replication success with respect to their ability to draw the correct inference when the underlying truth is known, while taking publication bias into account. Our results show that Bayesian metrics seem to slightly outperform frequentist metrics across the board. Generally, meta-analytic approaches seem to slightly outperform metrics that evaluate single studies, except in the scenario of extreme publication bias, where this pattern reverses. |
format | Online Article Text |
id | pubmed-8131945 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2021 |
publisher | The Royal Society |
record_format | MEDLINE/PubMed |
spelling | pubmed-81319452021-05-19 How best to quantify replication success? A simulation study on the comparison of replication success metrics Muradchanian, Jasmine Hoekstra, Rink Kiers, Henk van Ravenzwaaij, Don R Soc Open Sci Psychology and Cognitive Neuroscience To overcome the frequently debated crisis of confidence, replicating studies is becoming increasingly more common. Multiple frequentist and Bayesian measures have been proposed to evaluate whether a replication is successful, but little is known about which method best captures replication success. This study is one of the first attempts to compare a number of quantitative measures of replication success with respect to their ability to draw the correct inference when the underlying truth is known, while taking publication bias into account. Our results show that Bayesian metrics seem to slightly outperform frequentist metrics across the board. Generally, meta-analytic approaches seem to slightly outperform metrics that evaluate single studies, except in the scenario of extreme publication bias, where this pattern reverses. The Royal Society 2021-05-19 /pmc/articles/PMC8131945/ /pubmed/34017596 http://dx.doi.org/10.1098/rsos.201697 Text en © 2021 The Authors. https://creativecommons.org/licenses/by/4.0/Published by the Royal Society under the terms of the Creative Commons Attribution License http://creativecommons.org/licenses/by/4.0/ (https://creativecommons.org/licenses/by/4.0/) , which permits unrestricted use, provided the original author and source are credited. |
spellingShingle | Psychology and Cognitive Neuroscience Muradchanian, Jasmine Hoekstra, Rink Kiers, Henk van Ravenzwaaij, Don How best to quantify replication success? A simulation study on the comparison of replication success metrics |
title | How best to quantify replication success? A simulation study on the comparison of replication success metrics |
title_full | How best to quantify replication success? A simulation study on the comparison of replication success metrics |
title_fullStr | How best to quantify replication success? A simulation study on the comparison of replication success metrics |
title_full_unstemmed | How best to quantify replication success? A simulation study on the comparison of replication success metrics |
title_short | How best to quantify replication success? A simulation study on the comparison of replication success metrics |
title_sort | how best to quantify replication success? a simulation study on the comparison of replication success metrics |
topic | Psychology and Cognitive Neuroscience |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8131945/ https://www.ncbi.nlm.nih.gov/pubmed/34017596 http://dx.doi.org/10.1098/rsos.201697 |
work_keys_str_mv | AT muradchanianjasmine howbesttoquantifyreplicationsuccessasimulationstudyonthecomparisonofreplicationsuccessmetrics AT hoekstrarink howbesttoquantifyreplicationsuccessasimulationstudyonthecomparisonofreplicationsuccessmetrics AT kiershenk howbesttoquantifyreplicationsuccessasimulationstudyonthecomparisonofreplicationsuccessmetrics AT vanravenzwaaijdon howbesttoquantifyreplicationsuccessasimulationstudyonthecomparisonofreplicationsuccessmetrics |