Yuli Sutoto Nugroho, Siyou Li, Xiangyan Chen, Juexi Shao, Huanan Wu
This study investigates the reliability of large language models (LLMs) in scoring open-ended student essays across different academic domains and prompting conditions. We compare ten publicly available LLMs, including GPT-4o, Llama 3, Gemma, Mistral, DeepSeek, Qwen, and Phi-4, in their ability to evaluate student writing in Digital Literacy and Management courses. A total of 3476 scoring instances were generated by combining human ratings and LLM outputs under rubric-guided and unguided conditions. Evaluation metrics include inter-rater reliability using Cohen's Kappa, mean absolute error, Pearson correlation, and exact match rate. Results show that rubricguided prompts improved LLM agreement with human scores, reducing error and increasing alignment. Larger and instructiontuned models performed more reliably, although performance varied across academic subjects. While LLMs show promise as scalable tools to support educational assessment, they are not yet suitable to fully replace teacher judgment. These findings highlight the importance of structured prompting, model selection, and domain-specific calibration for the responsible use of LLMs in education. © 2025 IEEE.
Queen Mary University of London, Sch. of EECS, London, United Kingdom; Universitas Negeri Surabaya, Faculty of Engineering, Surabaya, Indonesia; Yunjing Intelligence Innovation Co., LTD., Shenzhen, China
