This study investigated how using different measurement models affects the ordering of respondents on personality measures and then how model choice affects the criterion-related validity of the measure. Of interest is what are called Generalized Graded Unfolding Models (GGUMs), which do not assume monotonically increasing Item Response Functions (in IRT), but instead require the response functions to form a single peak. It was hypothesized that these fairly new measurement models would more accurately estimate respondents' personalities when compared to models from general Item Response Theory (IRT), such as the Generalized Partial Credit Model (GPCM), as it was assumed that the GGUM has greater flexibility in modeling of the response process.
In addition, this study conducted impact analyses to assess the amount of rank order change that occurred at the upper end of scores on the personality measure. Criterion-related validities were not found to change much from measurement model to measurement model, but the impact analyses revealed substantial changes occurring at the upper end of the score distribution depending on the measurement model used. In a personnel selection context, this would result in the selection of different applicants when a top-down selection strategy is utilized.
Beyond possible linear relationships between personality and criteria, this study also investigated the possibility of non-linear relationships. More non-linear relations were observed with the GGUM compared to the GPCM.
Finally, a simulation comparing the GGUM to the GPCM was conducted to compare the accuracy of latent trait estimates from these models. Results found that item characteristics within a scale helped determine whether the GGUM or GPCM produced more accurate thetas and more accurate criterion-related validities from those thetas.
These findings suggest that it is important for researchers and practitioners to be aware of the characteristics of the items on their scales and use that knowledge to select the best measurement model. In addition, unfolding models may not accurately estimate thetas in all situations, particularly when the items in the scale do not exhibit meaningful unfolding. In these situations, if unfolding models were used, fairness and test efficacy could be in jeopardy.