A logical analysis of null hypothesis significance testing using popular terminology

BACKGROUND: Null Hypothesis Significance Testing (NHST) has been well criticised over the years yet remains a pillar of statistical inference. Although NHST is well described in terms of statistical models, most textbooks for non-statisticians present the null and alternative hypotheses (H(0) and H(A), respectively) in terms of differences between groups such as (μ(1) = μ(2)) and (μ(1) ≠ μ(2)) and H(A) is often stated to be the research hypothesis. Here we use propositional calculus to analyse the internal logic of NHST when couched in this popular terminology. The testable H(0) is determined by analysing the scope and limits of the P-value and the test statistic’s probability distribution curve. RESULTS: We propose a minimum axiom set NHST in which it is taken as axiomatic that H(0) is rejected if P-value< α. Using the common scenario of the comparison of the means of two sample groups as an example, the testable H(0) is {(μ(1) = μ(2)) and [([Formula: see text] (1) ≠ [Formula: see text] (2)) due to chance alone]}. The H(0) and H(A) pair should be exhaustive to avoid false dichotomies. This entails that H(A) is ¬{(μ(1) = μ(2)) and [([Formula: see text] (1) ≠ [Formula: see text] (2)) due to chance alone]}, rather than the research hypothesis (H(T)). To see the relationship between H(A) and H(T), H(A) can be rewritten as the disjunction H(A): ({(μ(1) = μ(2)) ∧ [([Formula: see text] (1) ≠ [Formula: see text] (2)) not due to chance alone]} ∨ {(μ(1) ≠ μ(2)) ∧ [[Formula: see text] (1) ≠ [Formula: see text] (2)) not due to (μ(1) ≠ μ(2)) alone]} ∨ {(μ(1) ≠ μ(2)) ∧ [([Formula: see text] (1) ≠ [Formula: see text] (2)) due to (μ(1) ≠ μ(2)) alone]}). This reveals that H(T) (the last disjunct in bold) is just one possibility within H(A). It is only by adding premises to NHST that H(T) or other conclusions can be reached. CONCLUSIONS: Using this popular terminology for NHST, analysis shows that the definitions of H(0) and H(A) differ from those found in textbooks. In this framework, achieving a statistically significant result only justifies the broad conclusion that the results are not due to chance alone, not that the research hypothesis is true. More transparency is needed concerning the premises added to NHST to rig particular conclusions such as H(T). There are also ramifications for the interpretation of Type I and II errors, as well as power, which do not specifically refer to H(T) as claimed by texts.