Changes in standard of candidates taking the MRCP(UK) Part 1 examination, 1985 to 2002: Analysis of marker questions

BACKGROUND: The maintenance of standards is a problem for postgraduate medical examinations, particularly if they use norm-referencing as the sole method of standard setting. In each of its diets, the MRCP(UK) Part 1 Examination includes a number of marker questions, which are unchanged from their use in a previous diet. This paper describes two complementary studies of marker questions for 52 diets of the MRCP(UK) Part 1 Examination over the years 1985 to 2001 to assess whether standards have changed. METHODS: Study 1, which used routinely collected information on the performance of 4405 marker items, used a statistical method to assess changes in performance across diets. Study 2 compared performances of individual candidates on 28 individual marker items that were shared by the 1996/2 and 2001/3 diets. RESULTS: Study 1 found evidence that candidate performance on the MRCP(UK) Part 1 Examination showed a gradual improvement over the period 1985 to 1997, which was followed by a sharp decline in performance until 2001. The 'dog-leg' in performance at 1997/3 was not an artefact of changed Examination Regulations, mix of UK and overseas candidates, or time from qualification until taking the Examination. Study 2 confirmed that performance in 2001/3 was significantly worse than in 1996/3, that the poorer performance was found in graduates of UK medical schools, and that candidates passing the Examination in 2001/3 performed less well than those passing in 1996/2. CONCLUSION: There has been a decline in the performance of graduates from UK medical schools taking the MRCP(UK) Part 1 examination. The reasons for this are not clear, but the finding has implications for medical education, and further studies are needed of performance in other postgraduate and undergraduate examinations. The use of norm-referencing as the sole method for setting the pass mark over this period meant that candidates passing the MRCP(UK) examination also had a lower standard. The MRCP(UK) Part 1 and Part 2 examinations now have their standard set by criterion-referencing.