Designing and validating a scale for evaluating the sources of unreliability of a high-stakes test

The idea of sources other than the test-takers’ knowledge leading to different results on high-stakes tests was the motif based on which the present investigation was initiated on the probable sources of unreliability of a test. For this purpose, the researchers went through a thorough literature review with the aim to identify the issues to be counted as sources of unreliability of a high-stakes test, i.e., the MA University Entrance Exam of English (UEEE) in Iran. First, 17 MA UEEE test-takers were asked to take part in a semi-structured interview to find out their ideas about such sources. The outcome of the thematic coding of the information from the literature and interviews was a 57-item Likert scale questionnaire which was reviewed by three assessment experts, revised accordingly, piloted with 57 MA UEEE test-takers, and revised again with 55 items remaining. The revised questionnaire was administered to 312 MA UEEE test-takers in Iran, and its reliability and construct validity were checked through Cronbach alpha (.89) and exploratory factor analysis, respectively. After checking its reliability and construct validity, 46 items remained and loaded on four factors which were named as the effect of test-takers (16 items), structure of the test and external concerns (13 items), administration conditions of the test (13 items), and role of proctors (4 items). The results of this study might familiarize test developers, test administrators, teachers, and test-takers with issues they should be aware of in developing or preparing for a high-stakes test like the MA UEEE.