Cargando…

Reanalyzing Head et al. (2015): investigating the robustness of widespread p-hacking

Head et al. (2015) provided a large collection of p-values that, from their perspective, indicates widespread statistical significance seeking (i.e., p-hacking). This paper inspects this result for robustness. Theoretically, the p-value distribution should be a smooth, decreasing function, but the d...

Descripción completa

Detalles Bibliográficos
Autor principal: Hartgerink, Chris H.J.
Formato: Online Artículo Texto
Lenguaje:English
Publicado: PeerJ Inc. 2017
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5337083/
https://www.ncbi.nlm.nih.gov/pubmed/28265523
http://dx.doi.org/10.7717/peerj.3068
_version_ 1782512330017341440
author Hartgerink, Chris H.J.
author_facet Hartgerink, Chris H.J.
author_sort Hartgerink, Chris H.J.
collection PubMed
description Head et al. (2015) provided a large collection of p-values that, from their perspective, indicates widespread statistical significance seeking (i.e., p-hacking). This paper inspects this result for robustness. Theoretically, the p-value distribution should be a smooth, decreasing function, but the distribution of reported p-values shows systematically more reported p-values for .01, .02, .03, .04, and .05 than p-values reported to three decimal places, due to apparent tendencies to round p-values to two decimal places. Head et al. (2015) correctly argue that an aggregate p-value distribution could show a bump below .05 when left-skew p-hacking occurs frequently. Moreover, the elimination of p = .045 and p = .05, as done in the original paper, is debatable. Given that eliminating p = .045 is a result of the need for symmetric bins and systematically more p-values are reported to two decimal places than to three decimal places, I did not exclude p = .045 and p = .05. I conducted Fisher’s method .04 < p < .05 and reanalyzed the data by adjusting the bin selection to .03875 < p ≤ .04 versus .04875 < p ≤ .05. Results of the reanalysis indicate that no evidence for left-skew p-hacking remains when we look at the entire range between .04 < p < .05 or when we inspect the second-decimal. Taking into account reporting tendencies when selecting the bins to compare is especially important because this dataset does not allow for the recalculation of the p-values. Moreover, inspecting the bins that include two-decimal reported p-values potentially increases sensitivity if strategic rounding down of p-values as a form of p-hacking is widespread. Given the far-reaching implications of supposed widespread p-hacking throughout the sciences Head et al. (2015), it is important that these findings are robust to data analysis choices if the conclusion is to be considered unequivocal. Although no evidence of widespread left-skew p-hacking is found in this reanalysis, this does not mean that there is no p-hacking at all. These results nuance the conclusion by Head et al. (2015), indicating that the results are not robust and that the evidence for widespread left-skew p-hacking is ambiguous at best.
format Online
Article
Text
id pubmed-5337083
institution National Center for Biotechnology Information
language English
publishDate 2017
publisher PeerJ Inc.
record_format MEDLINE/PubMed
spelling pubmed-53370832017-03-06 Reanalyzing Head et al. (2015): investigating the robustness of widespread p-hacking Hartgerink, Chris H.J. PeerJ Science Policy Head et al. (2015) provided a large collection of p-values that, from their perspective, indicates widespread statistical significance seeking (i.e., p-hacking). This paper inspects this result for robustness. Theoretically, the p-value distribution should be a smooth, decreasing function, but the distribution of reported p-values shows systematically more reported p-values for .01, .02, .03, .04, and .05 than p-values reported to three decimal places, due to apparent tendencies to round p-values to two decimal places. Head et al. (2015) correctly argue that an aggregate p-value distribution could show a bump below .05 when left-skew p-hacking occurs frequently. Moreover, the elimination of p = .045 and p = .05, as done in the original paper, is debatable. Given that eliminating p = .045 is a result of the need for symmetric bins and systematically more p-values are reported to two decimal places than to three decimal places, I did not exclude p = .045 and p = .05. I conducted Fisher’s method .04 < p < .05 and reanalyzed the data by adjusting the bin selection to .03875 < p ≤ .04 versus .04875 < p ≤ .05. Results of the reanalysis indicate that no evidence for left-skew p-hacking remains when we look at the entire range between .04 < p < .05 or when we inspect the second-decimal. Taking into account reporting tendencies when selecting the bins to compare is especially important because this dataset does not allow for the recalculation of the p-values. Moreover, inspecting the bins that include two-decimal reported p-values potentially increases sensitivity if strategic rounding down of p-values as a form of p-hacking is widespread. Given the far-reaching implications of supposed widespread p-hacking throughout the sciences Head et al. (2015), it is important that these findings are robust to data analysis choices if the conclusion is to be considered unequivocal. Although no evidence of widespread left-skew p-hacking is found in this reanalysis, this does not mean that there is no p-hacking at all. These results nuance the conclusion by Head et al. (2015), indicating that the results are not robust and that the evidence for widespread left-skew p-hacking is ambiguous at best. PeerJ Inc. 2017-03-02 /pmc/articles/PMC5337083/ /pubmed/28265523 http://dx.doi.org/10.7717/peerj.3068 Text en http://creativecommons.org/publicdomain/zero/1.0/ This is an open access article, free of all copyright, made available under the Creative Commons Public Domain Dedication (http://creativecommons.org/publicdomain/zero/1.0/) . This work may be freely reproduced, distributed, transmitted, modified, built upon, or otherwise used by anyone for any lawful purpose.
spellingShingle Science Policy
Hartgerink, Chris H.J.
Reanalyzing Head et al. (2015): investigating the robustness of widespread p-hacking
title Reanalyzing Head et al. (2015): investigating the robustness of widespread p-hacking
title_full Reanalyzing Head et al. (2015): investigating the robustness of widespread p-hacking
title_fullStr Reanalyzing Head et al. (2015): investigating the robustness of widespread p-hacking
title_full_unstemmed Reanalyzing Head et al. (2015): investigating the robustness of widespread p-hacking
title_short Reanalyzing Head et al. (2015): investigating the robustness of widespread p-hacking
title_sort reanalyzing head et al. (2015): investigating the robustness of widespread p-hacking
topic Science Policy
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5337083/
https://www.ncbi.nlm.nih.gov/pubmed/28265523
http://dx.doi.org/10.7717/peerj.3068
work_keys_str_mv AT hartgerinkchrishj reanalyzingheadetal2015investigatingtherobustnessofwidespreadphacking