Cargando…

Rater severity differences in English language as a second language speaking assessment based on rating experience, training experience, and teaching experience through many-faceted Rasch measurement analysis

Evaluating candidates’ answers in speaking skill is difficult and rarely explored. This task is challenging and can bring inconsistency in the rating quality among raters, especially in speaking assessments. Severe raters will bring more harm than good to the results that candidates receive. Many-fa...

Descripción completa

Detalles Bibliográficos
Autores principales: Mohd Noh, Muhamad Firdaus, Mohd Matore, Mohd Effendi Ewan
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Frontiers Media S.A. 2022
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9353031/
https://www.ncbi.nlm.nih.gov/pubmed/35936278
http://dx.doi.org/10.3389/fpsyg.2022.941084
_version_ 1784762784374849536
author Mohd Noh, Muhamad Firdaus
Mohd Matore, Mohd Effendi Ewan
author_facet Mohd Noh, Muhamad Firdaus
Mohd Matore, Mohd Effendi Ewan
author_sort Mohd Noh, Muhamad Firdaus
collection PubMed
description Evaluating candidates’ answers in speaking skill is difficult and rarely explored. This task is challenging and can bring inconsistency in the rating quality among raters, especially in speaking assessments. Severe raters will bring more harm than good to the results that candidates receive. Many-faceted Rasch measurement (MFRM) was used to explore the differences in teachers’ rating severity based on their rating experience, training experience, and teaching experience. The research uses a quantitative approach and a survey method to enlist 164 English teachers who teach lower secondary school pupils, who were chosen through a multistage clustered sampling procedure. All the facets involving teachers, candidates, items, and domains were calibrated using MFRM. Every teacher scored six candidates’ responses in a speaking test consisting of three question items, and they were evaluated across three domains, namely vocabulary, grammar, and communicative competence. Results highlight that the rating quality was different in terms of teachers’ rating experience and teaching experience. However, training experience did not bring any difference to teachers’ rating quality on speaking test. The evidence from this study suggests that the two main factors of teaching and rating experience must be considered when appointing raters for the speaking test. The quality of training must be improved to produce a rater with good professional judgment. Raters need to be supplied with answer samples with varied levels of candidates’ performance to practice before becoming a good rater. Further research might explore any other rater bias that may impact the psychological well-being of certain groups of students.
format Online
Article
Text
id pubmed-9353031
institution National Center for Biotechnology Information
language English
publishDate 2022
publisher Frontiers Media S.A.
record_format MEDLINE/PubMed
spelling pubmed-93530312022-08-06 Rater severity differences in English language as a second language speaking assessment based on rating experience, training experience, and teaching experience through many-faceted Rasch measurement analysis Mohd Noh, Muhamad Firdaus Mohd Matore, Mohd Effendi Ewan Front Psychol Psychology Evaluating candidates’ answers in speaking skill is difficult and rarely explored. This task is challenging and can bring inconsistency in the rating quality among raters, especially in speaking assessments. Severe raters will bring more harm than good to the results that candidates receive. Many-faceted Rasch measurement (MFRM) was used to explore the differences in teachers’ rating severity based on their rating experience, training experience, and teaching experience. The research uses a quantitative approach and a survey method to enlist 164 English teachers who teach lower secondary school pupils, who were chosen through a multistage clustered sampling procedure. All the facets involving teachers, candidates, items, and domains were calibrated using MFRM. Every teacher scored six candidates’ responses in a speaking test consisting of three question items, and they were evaluated across three domains, namely vocabulary, grammar, and communicative competence. Results highlight that the rating quality was different in terms of teachers’ rating experience and teaching experience. However, training experience did not bring any difference to teachers’ rating quality on speaking test. The evidence from this study suggests that the two main factors of teaching and rating experience must be considered when appointing raters for the speaking test. The quality of training must be improved to produce a rater with good professional judgment. Raters need to be supplied with answer samples with varied levels of candidates’ performance to practice before becoming a good rater. Further research might explore any other rater bias that may impact the psychological well-being of certain groups of students. Frontiers Media S.A. 2022-07-22 /pmc/articles/PMC9353031/ /pubmed/35936278 http://dx.doi.org/10.3389/fpsyg.2022.941084 Text en Copyright © 2022 Mohd Noh and Mohd Matore. https://creativecommons.org/licenses/by/4.0/This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
spellingShingle Psychology
Mohd Noh, Muhamad Firdaus
Mohd Matore, Mohd Effendi Ewan
Rater severity differences in English language as a second language speaking assessment based on rating experience, training experience, and teaching experience through many-faceted Rasch measurement analysis
title Rater severity differences in English language as a second language speaking assessment based on rating experience, training experience, and teaching experience through many-faceted Rasch measurement analysis
title_full Rater severity differences in English language as a second language speaking assessment based on rating experience, training experience, and teaching experience through many-faceted Rasch measurement analysis
title_fullStr Rater severity differences in English language as a second language speaking assessment based on rating experience, training experience, and teaching experience through many-faceted Rasch measurement analysis
title_full_unstemmed Rater severity differences in English language as a second language speaking assessment based on rating experience, training experience, and teaching experience through many-faceted Rasch measurement analysis
title_short Rater severity differences in English language as a second language speaking assessment based on rating experience, training experience, and teaching experience through many-faceted Rasch measurement analysis
title_sort rater severity differences in english language as a second language speaking assessment based on rating experience, training experience, and teaching experience through many-faceted rasch measurement analysis
topic Psychology
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9353031/
https://www.ncbi.nlm.nih.gov/pubmed/35936278
http://dx.doi.org/10.3389/fpsyg.2022.941084
work_keys_str_mv AT mohdnohmuhamadfirdaus raterseveritydifferencesinenglishlanguageasasecondlanguagespeakingassessmentbasedonratingexperiencetrainingexperienceandteachingexperiencethroughmanyfacetedraschmeasurementanalysis
AT mohdmatoremohdeffendiewan raterseveritydifferencesinenglishlanguageasasecondlanguagespeakingassessmentbasedonratingexperiencetrainingexperienceandteachingexperiencethroughmanyfacetedraschmeasurementanalysis