Multiple comparisons are common practice in statistical analysis. How to construct valid and powerful multiple testing procedures that control multiple testing error rates remains an interesting and challenging question.
Resampling methods and modeling are important tools to capturing the data structure, which makes construction of more powerful multiple testing procedures possible. In this work, we study the limitations of permutation tests as well as construct step-down short cut version of the partitioning test based on modeling in the discrete data analysis setting.
For example, in pharmacogenomics, multiple testing for significant association between genetic markers and phenotypes is of interest. Permuting response group labels to generate a reference distribution is often thought of as a convenient thresholding technique that automatically captures dependence in the data. It is shown in this work that, without non-trivial model assumptions,permutation testing may not control unconditional multiple testing error rates. We also show the lack of control for the conditional error rate.
When modeling is possible, analytical derivation of the joint distributions of the test statistics may be feasible. Upon satisfaction of certain sufficient conditions, we show how to
construct a more powerful step-down short-cut version of the partitioning test. Discussions and examples are within the context of logistic regression modeling for the independent binary outcome variable and GEE modeling for the correlated binary outcome variable.