The Use of an Automated Writing Evaluation System for Summative Assessment in an EFL Context: The Relationship Between Automated System Scores and Human Raters’ Scores

Proceedings of The 4th International Conference on Research in Teaching and Education

Year: 2021

DOI:

[Fulltext PDF]

The Use of an Automated Writing Evaluation System for Summative Assessment in an EFL Context: The Relationship Between Automated System Scores and Human Raters’ Scores

Hilal Yıldız, Safiye İpek Kuru Gönen

ABSTRACT: 

EFL students are constantly involved in evaluation of their writing skills to pass language tests and improve their writing in L2. Scoring individual essays and providing feedback may become an onerous task for teachers and they may need assistance in writing evaluation. As an outcome of recent technological advancements, Automated Writing Evaluation (AWE) systems have been utilized to assist teachers in scoring written essays and providing feedback. This study aims at investigating the relationship between teachers’ holistic scores and AWE holistic scores given to the same student essays. The participants of the study were seven (n=8) EFL instructors working at a university in Turkey. In line with the study’s aim, five students’ essays from each instructor’s class (n=35) were scored independently by the instructors, the AWE system, and another instructor who served as an external rater. A Spearman rank test was carried out and the findings revealed a positive and statistically significant correlation both between AWE scores and the instructors’ and between the scores of the instructors and the external rater. The findings indicate that the correlation between a human-rater and an AWE system can be compatible with the correlation between two human-raters. Moreover, the teachers noted that there were similarities between their assessment and the AWE assessment in terms of the similarity of the scores, the score intervals, and the rationales for the scores in each interval. All in all, the findings may contribute to a better understanding of using these systems in classroom-based writing assessment in EFL contexts.

Keywords: automated writing evaluation; automated essay scoring; computer-assisted writing evaluation; writing assessment in EFL classes; writing in L2.