Skip to Main Content
 

Global Search Box

 
 
 
 

Files

ETD Abstract Container

Abstract Header

Issues of reliability and validity in using portfolio assessment to measure foreign language teacher performance

Gelinas, Anna-Marie

Abstract Details

1998, Doctor of Philosophy, Ohio State University, ED Policy and Leadership.

Portfolio assessment is an increasingly common trend in teacher education programs because the multifaceted nature of portfolios allows for the demonstration of a variety of strengths and competencies essential to effective teaching. Portfolios provide preservice teachers a means with which they can demonstrate their conceptual knowledge and their ability to apply that knowledge in the classroom (pedagogical content knowledge). The reliability and validity of the portfolio scores in making professional decisions concerning the competence of preservice teachers, however, is a difficult issue because of the variability found among individual portfolios.

The current study investigated issues of reliability and validity in using portfolios to evaluate the pedagogical abilities of preservice teachers in an elementary foreign language methods course. A correlational analysis among holistic portfolio scores and other measures used to evaluate students in the course demonstrated a positive relationship between portfolio scores and two of the independent variables: performance-based tasks (r=.47) and years of teaching experience (r=.35). The regression analysis indicated that the performance tasks were the best predictor of portfolio scores (22% of the variance explained) followed by years of experience (16%). These results established the predictive and criterion-related validity of the portfolios as an accurate method of measuring pedagogical content knowledge in the course. A principal components factor analysis was conducted on the holistic data to examine construct validity. The results yielded two factors with eigenvalues of 2.37 (conceptual knowledge) and 1.14 (pedagogical content knowledge), and explained 70% of the variance in the instruments. The final exam, GPA, and years of experience loaded on factor one; portfolios and tasks loaded on factor two.

Inter-rater reliability and score reliability were investigated through a Rasch model FACETS analysis and a generalizability study to determine the relative impact of factors such as raters and items on the portfolio scores. The level of rater agreement (r=.81) and the score reliability (r=.90) established by the FACETS analysis showed that the portfolios were rated consistently and provided an accurate measure of teaching competency. The results of the generalizability study indicated that while the rater and item facets alone did not account for a large portion of the variance (1% of total variance for each facet), the item by person (12%) and the rater by person interactions (36%) did account for a larger portion. It was concluded that these interactions might have been due to the individuality and format of the portfolios when combined with the personalities and preferences of the raters. The subjectivity inherent in rating portfolios therefore continues to be a limitation in the implementation of portfolio assessment as well as an area for further research.

Instructing preservice teachers to implement portfolio assessment through personal experience with portfolios may foster their use of portfolios in the future. Subjects were surveyed about their understanding of portfolio assessment and rating scales previous to the course and after developing a portfolio. The results showed that the students had an increased understanding of the nature of portfolio assessment after their experience with portfolios in the course. The students also demonstrated a positive attitude toward portfolios and a willingness to use them with future students.

Gilbert Jarvis (Advisor)
Ayres D'Costa (Advisor)
Cora Kaylani (Committee Member)
170 p.

Recommended Citations

Citations

  • Gelinas, A.-M. (1998). Issues of reliability and validity in using portfolio assessment to measure foreign language teacher performance [Doctoral dissertation, Ohio State University]. OhioLINK Electronic Theses and Dissertations Center. http://rave.ohiolink.edu/etdc/view?acc_num=osu1248453157

    APA Style (7th edition)

  • Gelinas, Anna-Marie. Issues of reliability and validity in using portfolio assessment to measure foreign language teacher performance. 1998. Ohio State University, Doctoral dissertation. OhioLINK Electronic Theses and Dissertations Center, http://rave.ohiolink.edu/etdc/view?acc_num=osu1248453157.

    MLA Style (8th edition)

  • Gelinas, Anna-Marie. "Issues of reliability and validity in using portfolio assessment to measure foreign language teacher performance." Doctoral dissertation, Ohio State University, 1998. http://rave.ohiolink.edu/etdc/view?acc_num=osu1248453157

    Chicago Manual of Style (17th edition)