The Type I error rate inflates greatly when multiple hypotheses are tested using an unadjusted alpha per test procedure. Therefore, several alpha-adjustment Multiple Hypothesis Testing (MHT) procedures can be used to control the Type I error inflation while providing adequate statistical power. There are numerous statistical designs that involve MHT, such as factorial ANOVA. This study investigated the Type I error rates and statistical power rates of several alpha-adjustment MHT procedures (Bonferroni, Holm, Hochberg, and Benjamini-Hochberg (B-H)) in a balanced factorial ANOVA. Three indicators for Type I error rates were used: samplewise familywise error rate (SFWER), testwise familywise error rate (TFWER), and false discovery rate (FDR). Three criteria for statistical power rates were employed: samplewise power (SPOWER), testwise power (TPOWER), and true discovery rate (TDR). MHT procedures were also compared to the unadjusted alpha per test procedure. All statistical analyses were done with 20,000 replications as a Monte Carlo simulation in the R programming language.
Two-way and three-way fixed-effects balanced designs were analyzed. The sample size per cell was 32 in the two-way and 16 for the three-way. A medium effect size of .50 for all false null effects was used to create data with different patterns of means. MHT procedures were found to have advantages over the unadjusted alpha per test procedure in terms of controlling the Type I error inflation at an accurate level. Specifically, Bonferroni, Holm, and Hochberg were better able to control the Type I error rates at .05. The SFWER and FDR from the B-H procedure inflate under certain conditions. The Bonferroni procedure has the lowest power while the B-H procedure has the greatest power. The Hochberg procedure worked best in this study overall. However, if the independence assumption is not met, the Holm procedure can be the best choice.