Single-case designs (SCDs) are often used to examine the impact of an intervention over brief periods of time (Kratochwill & Stoiber, 2002; Segool, Brinkman, & Carlson, 2007). The majority of SCDs are inspected using visual analysis (Kromrey & Foster-Johnson, 1996; Morgan & Morgan, 2009). Although the single-case literature suggests that visual analyses have merit (Brossart, Parker, Olson, & Mahadevan, 2006; Kratochwill & Brody, 1978) there are concerns regarding the reliability of the procedure (Shadish et al., 2009). Recent advances in hierarchical linear models (HLM) allow for statistical analyses of treatment effects (Nagler, Rindskopf, & Shadish, 2008), thus making it possible to compare and contrast results from HLM and visual analyses to ascertain if the different methods yield consistent conclusions. This work performed a series of sensitivity analyses while also exploring ways in which HLM can be used to examine new and different questions when dealing with published single-case data.
The work applied analyses to ABAB designs only. In addition to reporting the results of visual analysis performed by the original authors, it also utilized recently published guidelines by the What Works Clearinghouse (WWC) that standardize visual analysis processes (Kratochwill, Hitchcock, Horner, Levin, Odom, Rindskopf, & Shadish, 2010). The comparisons presented here are based on nine, single-case studies that meet WWC design standards. All studies examined intervention impacts on behavioral outcomes. UnGraph digitizing software was used to quantify results from ABAB graphs and HLM and STATA software were used to perform statistical analyses. In addition to applying a statistical procedure to check conclusions about treatment effects based on visual analyses, HLM was used to examine between-subject variation of performance on outcome measures. In order to statistically describe treatment impacts, effect size estimates were calculated using four methods: (a) the percentage of nonoverlapping data (Morgan & Morgan, 2009), (b) the Standardized Mean Difference (Busk & Serlin, 1992), (c) the improvement rate difference and (d) R2, in order to assess the proportion variance in the dependent variable that can be explained by treatment exposure.