Evaluating Student Communication Skills and User Acceptability of an NLP-Based Automated Essay Grading System

Abstract

This study examined the effectiveness, user experience, and acceptability of a self–developed Natural Language Processing (NLP)-based automated essay grading (AEG) system in evaluating students' communication skills. The system was tested with 35 IT students and 16 IT teachers from Central Bicol State University of Agriculture – Sipocot, College of Information Technology, to assess its reliability and usability. Results showed that the system provided consistent and objective evaluations, with a Cohen’s kappa value of 0.797, indicating substantial agreement with human grading. Teachers found the system helpful for reducing their grading workload and appreciated its reliability, while students valued the constructive feedback it provided to improve their writing. Both groups rated the system highly for its accuracy, consistency, and fairness. However, the system showed some limitations in assessing more nuanced aspects of communication, such as coherence and relevance, which should be addressed in future enhancements. Overall, this study highlights the potential of AEG systems to support education by providing reliable, efficient, and actionable assessments of communication skills.