ChatGPT: is it reliable as an automated writing evaluation tool?

Abstract

This study primarily aims to give an understanding of whether or not teachers could rely on AI technology, specifically ChatGPT, to score students’ writings. The study was conducted with the participation of EFL university students. The students were assigned different writing tasks for five weeks, and the tasks were scored by a teacher and ChatGPT separately. Then, their scores were compared to see the extent to which ChatGPT and teacher scores differed on the SPSS. The test results indicated no statistically significant differences in the scores the bot or the teacher gave. Additionally, the results were supported by the qualitative analysis of the teachers’ perception of ChatGPT use for automated writing evaluation. The teachers’ perceptions indicated their positive attitudes towards its use for the evaluation process and general use for enhancing instruction and learning, together with the concerns and suggestions to make the most of ChatGPT. The study gives insights into the integration of ChatGPT into the assessment process and its effectiveness for class practices.