This study was designed to evaluate the performance of missing data treatments with restrictive and inclusive strategies in a two-level hierarchical linear model with missing at random (MAR) missingness in terms of bias, Root Mean Square Error (RMSE), and width and coverage rate of confidence interval. The missing data treatments included in this study were listwise deletion, mean substitution, restrictive and inclusive EM, restrictive and inclusive multiple imputation (MI). The number of level-2 predictors, proportion of missingness (PM) and sample size (N) were manipulated as study factors.
The number of level-2 predictors and sample size appeared not to have a distinct impact on the performance of missing data treatments for level-2 missing data in terms of bias. However, the proportion of missing data significantly tends to affect the performance of missing data treatments with large effect so that with larger proportion of missingness, the relative bias difference among missing data treatments tends to increase in most fixed effects and some random effects.
Inclusive MI and listwise deletion generally outperformed the other missing data treatments producing practically acceptable bias in most fixed effects that are highly related to missingness. Restrictive EM and inclusive EM also performed well with some exceptions with large proportion of missingness (PM=30%). Restrictive MI and mean substitution produced unacceptable bias even with smaller proportions of missingness (PM=5% or 15%). For random effects, every missing data treatment was effective except for the non-significant Tau11.
Listwise deletion tends to provide the largest RMSE on both fixed and random effects. The relative difference in the RMSE between listwise deletion and the other missing data treatments was substantially large with large proportion of missingness (PM=30%) and smaller sample sizes (N<80 or 40).
Furthermore, listwise deletion provided the largest confidence intervals for both fixed and random effects. Again, the difference in the confidence interval width between listwise deletion and the other missing data treatments was substantially large with smaller sample sizes (N<80 or 40) and large proportion of missingness (PM=30%). The confidence interval coverage rates of mean substitution and inclusive EM were problematic with short confidence intervals for fixed effects when proportions of missingness are larger (PM>15% or 30%). Listwise deletion and inclusive EM also provided poor confidence interval coverage on Tau00 when missingness is large (PM=30%) and sample size is small (N=40).
Therefore, inclusive MI and restrictive EM may be a viable option with MAR missingness at level 2 in HLM to applied researchers. However, inclusive MI is preferred when proportion of missing data is large (PM=30%). Finally, it should be noted that no missing data treatment was effective on non-significant fixed or random effects smaller than .30.