Misinterpretations of P-values and statistical tests persists among researchers and professionals working with statistics and epidemiology

BACKGROUND: The aim was to investigate inferences of statistically significant test results among persons with more or less statistical education and research experience. METHODS: A total of 75 doctoral students and 64 statisticians/epidemiologist responded to a web questionnaire about inferences of statistically significant findings. Participants were asked about their education and research experience, and also whether a ‘statistically significant’ test result (P = 0.024, α-level 0.05) could be inferred as proof or probability statements about the truth or falsehood of the null hypothesis (H(0)) and the alternative hypothesis (H(1)). RESULTS: Almost all participants reported having a university degree, and among statisticians/epidemiologist, most reported having a university degree in statistics and were working professionally with statistics. Overall, 9.4% of statisticians/epidemiologist and 24.0% of doctoral students responded that the statistically significant finding proved that H(0) is not true, and 73.4% of statisticians/epidemiologists and 53.3% of doctoral students responded that the statistically significant finding indicated that H(0) is improbable. Corresponding numbers about inferences about the alternative hypothesis (H(1)) were 12.0% and 6.2% about proving H(1) being true and 62.7 and 62.5% for the conclusion that H(1) is probable. Correct inferences to both questions, which is that a statistically significant finding cannot be inferred as either proof or a measure of a hypothesis’ probability, were given by 10.7% of doctoral students and 12.5% of statisticians/epidemiologists. CONCLUSIONS: Misinterpretation of P-values and statistically significant test results persists also among persons who have substantial statistical education and who work professionally with statistics.