The impact of automated writing evaluation on second language writing skills of Chinese EFL learners: a randomized controlled trial

INTRODUCTION: In the context of the burgeoning field of second language (L2) education, where proficient writing plays an integral role in effective language acquisition and communication, the ever-increasing technology development has influenced the trajectory of L2 writing development. METHODS: To address the need for enhanced writing skills among English as a Foreign Language (EFL) learners, this study investigates the efficacy of Automated Writing Evaluation (AWE) training. A randomized controlled trial employing repeated measures was conducted, involving a participant pool of 190 Chinese EFL students. The study comprehensively assessed the effects of AWE training, utilizing the Grammarly platform—an AI-driven program—on various dimensions of writing skills, encompassing task achievement, coherence and cohesion, lexicon, and grammatical accuracy. Control variables included writing self-efficacy and global English proficiency. Writing skills were evaluated through the administration of an International English Language Testing System (IELTS) writing sample test. RESULTS: The results unequivocally demonstrate that the experimental group consistently exhibited superior performance across all facets of writing skills compared to the control group. Furthermore, the predictive influence of pre-test scores was pronounced in task achievement, coherence and cohesion, and lexicon, highlighting the pivotal role of learners’ initial proficiency levels in shaping subsequent writing outcomes. Notably, the emergence of writing self-efficacy as a significant predictor of task achievement and coherence and cohesion underscores the role of learners’ beliefs and confidence in shaping their writing abilities. DISCUSSION: These findings conclusively suggest that Artificial Intelligence-based instructional programs, specifically AWE, hold the potential to effectively enhance second language writing skills, especially among learners with lower proficiency levels. This study carries crucial implications for EFL educators and researchers, advocating for the seamless integration of AWE into pedagogical strategies to foster a marked improvement in writing competence.