Skip to Main Content
 

Global Search Box

 
 
 
 

ETD Abstract Container

Abstract Header

Assessing the Absolute and Relative Performance of IRTrees Using Cross-Validation and the RORME Index

DiTrapani, John B

Abstract Details

2019, Doctor of Philosophy, Ohio State University, Psychology.
This dissertation introduces a new model evaluation tool - the RORME index - that can be used to select an item response model among competing alternatives. This criterion assesses the out-of-sample predictive performance of candidate models using a k-fold cross-validation procedure. The validity of RORME is evaluated with several simulation studies, which conclude that the proposed index performs well under a multitude of conditions. RORME often follows a similar selection pattern as the AIC; however, unlike the AIC or BIC, it does not rely on underlying model assumptions or likelihood-based estimation. The RORME index is also flexible in how a researcher would prefer a model to be evaluated - the speci c performance metric used to calculate RORME can be changed and investigations into local mis fit (e.g. item- or person-speci c mis t) can be conducted. The RORME index is of particular interest when utilized to compare item response tree, or IRTree, models with non-IRTree alternative models. IRTree models are a particular application of the item response theory framework that allow interesting, novel research questions to be addressed. One of these novel explorations is assessing the most appropriate process that a respondent undertakes when responding to an item. This type of research question naturally requires comparing several candidate models and eventually endorsing the most appropriate alternative. Traditional model selection criteria, such as AIC or BIC, may not be appropriate for these types of model comparisons, since IRTree models require the underlying data to be transformed. In this project, the RORME index is developed and applied to directly compare IRTree and non-IRTree models. Simulation results suggest that the new metric can successfully determine the appropriate model when applied in this context, even when criteria like the AIC or BIC are invalid.
Paul De Boeck (Advisor)
Andrew Hayes (Committee Member)
Jolynn Pek (Committee Member)
198 p.

Recommended Citations

Citations

  • DiTrapani, J. B. (2019). Assessing the Absolute and Relative Performance of IRTrees Using Cross-Validation and the RORME Index [Doctoral dissertation, Ohio State University]. OhioLINK Electronic Theses and Dissertations Center. http://rave.ohiolink.edu/etdc/view?acc_num=osu1555328378474406

    APA Style (7th edition)

  • DiTrapani, John. Assessing the Absolute and Relative Performance of IRTrees Using Cross-Validation and the RORME Index. 2019. Ohio State University, Doctoral dissertation. OhioLINK Electronic Theses and Dissertations Center, http://rave.ohiolink.edu/etdc/view?acc_num=osu1555328378474406.

    MLA Style (8th edition)

  • DiTrapani, John. "Assessing the Absolute and Relative Performance of IRTrees Using Cross-Validation and the RORME Index." Doctoral dissertation, Ohio State University, 2019. http://rave.ohiolink.edu/etdc/view?acc_num=osu1555328378474406

    Chicago Manual of Style (17th edition)