Murine genetic models of obesity: type I error rates and the power of commonly used analyses as assessed by plasmode-based simulation

BACKGROUND/OBJECTIVES: Genetic contributors to obesity are frequently studied in murine models. However, the sample sizes of these studies are often small, and the data may violate assumptions of common statistical tests such as normality of distributions. We examined whether, in these cases, type I error rates and power are affected by the choice of statistical test. SUBJECTS/METHODS: We conducted “plasmode”-based simulation using empirical data on body mass (weight) from murine genetic models of obesity. For the type I error simulation, the weight distributions were adjusted to ensure no difference in means between control and mutant groups. For the power simulation, the distributions of the mutant groups were shifted to ensure specific effect sizes. Three to 20 mice were resampled from the empirical distributions to create a plasmode. We then computed type I error rates and power for five common tests on the plasmodes: Student’s t test, Welch’s t test, Wilcoxon rank sum test (aka, Mann-Whitney U test), permutation test, and bootstrap test. RESULTS: We observed type I error inflation for all tests, except the bootstrap test, with small samples (≤5). Type I error inflation decreased as sample size increased (≥8) but remained. The Wilcoxon test should be avoided because of heterogeneity of distributions. For power, a departure from the reference was observed with small samples for all tests. Compared with the other tests, the bootstrap test had less power with small samples. CONCLUSIONS: Overall, the bootstrap test is recommended for small samples to avoid type I error inflation, but this benefit comes at the cost of lower power. When sample size is large enough, Welch’s t test is recommended because of high power with minimal type I error inflation.