One challenge facing advocates of guided assessment is the development of evaluative criteria that represent clear, significant, and useful levels of expertise. This dissertation develops and tests a guided assessment for evaluating mathematical problem-solving called the [A]ttention, [T]actics, [O]perations, [S]olutions, and [U]nique-conditions model. The five ATOSU attributes measure a hierarchical levels of both conceptual and procedural thinking viewed in mathematical problem-solving. Levels of the students' goal-directed verbal, visual, and mathematical activity are judged and rated under a limited number of problem conditions. The ATOSU approach asks teachers to use probing questions to direct problem-solving activity, to cue students when they are confused, and to analogically relate one mathematical topic to another mathematical topic or to a variety of "real world" topics. These criteria or rated attributes are considered useful when they can communicate a range of quality levels that, when analyzed by teachers and parents in decision making, provide insights into learning and feedback that increases performance. Although one study cannot possibly complete the task of validating such a model, this study makes some preliminary attempts at evaluating the quality of these attributes as clear benchmarks of conceptual and procedural growth.
Three sets of research questions are addressed using measurement methods, experimental methods, and qualitative methods. Since the ATOSU model is necessarily complex, it needs to be developed in three stages, answering one set of questions at each stage. In the first stage, the focus of this research is on the validity and reliability of the predicted conceptual and procedural hierarchies. In the second stage, a 2 x 3 factorial design compares the unique contributions of gender and three different treatments (Test-only, ATOSU assessment, and Tutored-only) on conceptual, procedural, and problem-solving scales of National Assessment of Educational Progress (NAEP). In the final stage, snippets of qualitative data and the students responses are used to illustrate some major themes relative to the ATOSU criteria and whether this qualitative data supports the model.
Correlations of the five attributes suggested a simplex pattern that is indicative of a hierarchy, but these correlations may have been influenced by the dependence among ratings resulting from pooling these ratings across judges when calculating the correlation matrix. Measurement results indicated that the ATOSU attributes fit the Many-Facets Rasch model, and indicated a hierarchical relationship among attributes for all but the [U] attribute. Student, task, judge, and attribute facets all demonstrated reliabilities coefficients over 0.90; however, inter-rater reliabilities computed using Guilford's method varied from moderately high to low for each attribute.
Experimental results yielded a multivariate interaction effect between levels of gender with levels of treatment on the conceptual, procedural, and problem-solving dependent variables. Graphs of the interactions were each interpreted for the conceptual, procedural, and problem-solving scales of the NAEP. A large interaction effect on the univariate test on the conceptual scale seemed to explain most of the multivariate interaction. Simple effects measured by these tests revealed large advantages in conceptual performance favoring males in the Test-only group; however, ATOSU females had significantly higher NAEP scores when compared to the Test-only and Tutored-only females.
Qualitative data was finally used to illustrate some major themes and demonstrate how the ATOSU attributes were judged. Such illustrations are left for the reader to accept or reject.