Language practice that includes meaningful interaction is a critical component of many current teaching theories. At the same time, existing research on intelligent computer-aided language learning (ICALL) systems has focused primarily on providing practice with grammatical forms. Thus, there is a real need for ICALL systems that provide accurate content assessment. This thesis addresses that need by taking an empirically driven approach to the exploration of content assessment.
The primary source of material for this exploration is a new corpus of language learner data. The corpus is comprised exclusively of responses to short-answer reading comprehension questions by intermediate English language learners. Responses to these questions are ideal for developing and testing an approach to content error diagnosis because they exhibit linguistic variation on lexical, morphological, syntactic and semantic levels, but they have definable target responses that reflect acceptable content. The corpus is one of the first known to be annotated with diagnoses of meaning errors. Diagnoses were developed by analyzing the learner data and adopting an annotation scheme based on target modification. This corpus provided invaluable insight into the considerations necessary for developing an approach to diagnosing meaning errors.
Because variation is possible across learner responses, any degree of content assessment must be flexible and support the comparison of target and learner responses on several levels including token, chunk and relation levels. This thesis presents an architecture for a content assessment module (CAM) which provides this flexibility using multiple, surface-based matching strategies and existing language processing tools. Results show that content assessment using shallow NLP strategies is feasible for language activities of the type used for the case study. Detection of meaning errors approaches 90%. Results also indicate that diagnosis of meaning errors is possible using an approach that relies on machine learning, though additional testing with a larger corpus is needed. By developing and testing this model, as well as exploring the middle ground of activities, this work begins to bridge the gap between what is practical and achievable from a processing perspective and what is desirable from the perspective of current theories of language instruction.