Yeni Anistyasari, Shintami C. Hidayati, Suparji Suparji, Ekohariadi Ekohariadi, Dyah Ayu Kusumaningtyas
This study aimed to examine the reliability and accuracy of ChatGPT in scoring reflective essays. The study took place in a course on Teaching and Learning in English for Academic Writing. Students completed a task to write a systematic literature review on computing education. The main goal was to compare ChatGPT scores with scores from human tutors. ChatGPT received assessment rubrics and prompt instructions. The research team verified ChatGPT's understanding of the rubric. ChatGPT then scored the essays independently. The research used 72 student reflections as the dataset. Expert tutors also scored the same essays. The comparison used statistical tests. Cohen's Kappa and Intraclass Correlation Coefficient (ICC) showed low agreement. The results indicated poor consistency between ChatGPT and human scores. Error analysis showed a consistent pattern. ChatGPT gave higher scores than human raters. The difference was statistically significant. The study found that ChatGPT worked efficiently and gave fast results. However, ChatGPT lacked the ability to interpret subtle meaning. ChatGPT also did not always follow the rubric as expected. The study concludes that ChatGPT can support formative feedback. However, ChatGPT should not replace human judgment in academic writing assessment. © The Authors, published by EDP Sciences, 2025.
Faculty of Engineering, Universitas Negeri Surabaya, Surabaya, 60231, Indonesia; Department of Informatics, Institut Teknologi Sepuluh Nopember, Surabaya, 60111, Indonesia; Department of Business and Management, Southern Taiwan University of Science and Technology, Tainan City, 71005, Taiwan
Research at a Glance
Register to unlock
Topics & SDG Alignment
Register to unlock
Collaboration
Register to unlock
Author Profile (Selected)
Register to unlock
References Overview
Register to unlock
Journal & Source
Register to unlock
Metadata & Integrity
Register to unlock