A Framework to Avoid Significance Fallacy

This manuscript presents a concise approach to tackle the widespread misuse of statistical significance in scientific research, focusing on public health. It offers practical guidance for conducting accurate statistical evaluations and promoting easily understandable results based on actual evidence. When conducting a statistical study to inform decision-making, it is recommended to follow a step-by-step sequence while considering various factors. Firstly, multiple target hypotheses should be adopted to assess the compatibility of experimental data with different models. Reporting all P-values in full, rounded in order to have a single non-zero significant digit, enhances transparency and reduces the likelihood of exaggerating the state of the evidence. Detailed documentation of the procedures used to evaluate the compatibility between test assumptions and data should be provided for rigorous assessment. A descriptive evaluation of results can be aided by using statistical compatibility ranges, which help avoid misrepresenting the evidence. Separately evaluating and reporting statistical compatibility and effect size prevents the magnitude fallacy. Additionally, reporting measures of statistical effect size enables evaluation of sectoral relevance, such as clinical significance. Multiple compatibility intervals, such as 99%, 95%, and 90% confidence intervals, should be reported to allow readers to assess the variation of P-values based on the width of the interval. These recommendations aim to enhance the robustness and interpretability of statistical analyses and promote transparent reporting of findings. The author encourages journal adoption of similar frameworks to enhance scientific rigor, particularly in the field of medical science.