Skip to Main Content
 

Global Search Box

 
 
 
 

Files

File List


Dissertation.pdf (2.39 MB)

ETD Abstract Container

Abstract Header

Assessing the Validity of a Measure of the Culture of Evidence at Two-Year Colleges

Wallace-Pascoe, Dawn Marie
http://rave.ohiolink.edu/etdc/view?acc_num=osu1373305560

Abstract Details

2013, Doctor of Philosophy, Ohio State University, EDU Policy and Leadership.
This dissertation explores the validity of a measure designed to assess the extent to which two-year community and technical colleges have a culture of evidence. Culture of evidence refers to an institutional culture in which decisions related to student outcomes are based on evidence and data. The expected result of developing a culture of evidence is that colleges will become more successful at identifying “what works” in efforts to improve student success, which ultimately translates to better outcomes for students. The idea of developing a culture of evidence has gained prominence in higher education in recent years, prompted by increased calls for accountability in education in general and by the implementation of national initiatives specifically aimed at improving outcomes at two-year colleges. While the idea of building a culture of evidence has received attention in the literature, little has been done to measure the concept quantitatively. This study addressed this gap by developing, pilot testing, and conducting a survey designed to measure the extent to which two-year colleges have a culture of evidence. The process used to develop the survey included a literature review, think-alouds with institutional researchers, and expert content review. The online survey was administered to 853 directors of institutional research or institutional effectiveness identified through a query of public, two-year, associate’s degree-granting institutions. A total of 369 individuals responded. The framework for conducting the validation study was the unified conceptualization of validity from the American Psychological Association, American Educational Research Association, and National Council on Measurement in Education’s Standards for Educational and Psychological Testing (1999), in tandem with validation methods recommended by Allen and Yen (2002). Following this framework, five aspects of validity were assessed: (a) content, (b) response process/substantive, (c) internal structure, (d) predictive/relationships to other variables, and (e) consequential validity. Data analyses included descriptive statistics, reliability, analysis of variance, principal components analysis, exploratory and confirmatory factor analyses, and structural equation modeling. The results indicated strong evidence of content, response process/substantive, and consequential validity. However, the results for internal structural and expected relationships validity were more mixed. The number of full-time equivalent (FTE) staff who work in an institutional research function and membership in the Achieving the Dream initiative had statistically significant and positive relationships to the culture of evidence, but expected relationships to other contextual variables such as institution size were not found. When treated as an aggregate measure, a college’s total culture of evidence score also predicted the extent to which colleges engaged in institutional data practices that are consistent with having a culture evidence such as using student outcomes data in decision-making processes and faculty requesting data for decisions. A single study cannot determine validity-- validity should involve ongoing research to gather evidence for the interpretation of the scores for the intended use of a test or instrument (Cronbach and Meehl, 1955; Kane, 2006). This dissertation demonstrated a first attempt at gathering validity evidence for a new measure that could assist two-year colleges in their efforts to implement changes to improve student outcomes.
Ann O'Connell (Advisor)
Julia Carpenter-Hubin (Committee Member)
Jerome D'Agostino (Committee Member)
Dorinda Gallant (Committee Member)
170 p.
Educational Evaluation; Educational Tests and Measurements; Higher Education; Higher Education Administration
Culture of evidence; two-year colleges; accountability; institutional research; institutional effectiveness

Recommended Citations

Citations

  • Wallace-Pascoe, D. M. (2013). Assessing the Validity of a Measure of the Culture of Evidence at Two-Year Colleges [Doctoral dissertation, Ohio State University]. OhioLINK Electronic Theses and Dissertations Center. http://rave.ohiolink.edu/etdc/view?acc_num=osu1373305560

    APA Style (7th edition)

  • Wallace-Pascoe, Dawn. Assessing the Validity of a Measure of the Culture of Evidence at Two-Year Colleges. 2013. Ohio State University, Doctoral dissertation. OhioLINK Electronic Theses and Dissertations Center, http://rave.ohiolink.edu/etdc/view?acc_num=osu1373305560.

    MLA Style (8th edition)

  • Wallace-Pascoe, Dawn. "Assessing the Validity of a Measure of the Culture of Evidence at Two-Year Colleges." Doctoral dissertation, Ohio State University, 2013. http://rave.ohiolink.edu/etdc/view?acc_num=osu1373305560

    Chicago Manual of Style (17th edition)

osu1373305560
650
© 2013, all rights reserved.
This open access ETD is published by The Ohio State University and OhioLINK.