Cargando…

Construct-level predictive validity of educational attainment and intellectual aptitude tests in medical student selection: meta-regression of six UK longitudinal studies

BACKGROUND: Measures used for medical student selection should predict future performance during training. A problem for any selection study is that predictor-outcome correlations are known only in those who have been selected, whereas selectors need to know how measures would predict in the entire...

Descripción completa

Detalles Bibliográficos
Autores principales: McManus, IC, Dewberry, Chris, Nicholson, Sandra, Dowell, Jonathan S, Woolf, Katherine, Potts, Henry WW
Formato: Online Artículo Texto
Lenguaje:English
Publicado: BioMed Central 2013
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3827328/
https://www.ncbi.nlm.nih.gov/pubmed/24229353
http://dx.doi.org/10.1186/1741-7015-11-243
_version_ 1782291060683177984
author McManus, IC
Dewberry, Chris
Nicholson, Sandra
Dowell, Jonathan S
Woolf, Katherine
Potts, Henry WW
author_facet McManus, IC
Dewberry, Chris
Nicholson, Sandra
Dowell, Jonathan S
Woolf, Katherine
Potts, Henry WW
author_sort McManus, IC
collection PubMed
description BACKGROUND: Measures used for medical student selection should predict future performance during training. A problem for any selection study is that predictor-outcome correlations are known only in those who have been selected, whereas selectors need to know how measures would predict in the entire pool of applicants. That problem of interpretation can be solved by calculating construct-level predictive validity, an estimate of true predictor-outcome correlation across the range of applicant abilities. METHODS: Construct-level predictive validities were calculated in six cohort studies of medical student selection and training (student entry, 1972 to 2009) for a range of predictors, including A-levels, General Certificates of Secondary Education (GCSEs)/O-levels, and aptitude tests (AH5 and UK Clinical Aptitude Test (UKCAT)). Outcomes included undergraduate basic medical science and finals assessments, as well as postgraduate measures of Membership of the Royal Colleges of Physicians of the United Kingdom (MRCP(UK)) performance and entry in the Specialist Register. Construct-level predictive validity was calculated with the method of Hunter, Schmidt and Le (2006), adapted to correct for right-censorship of examination results due to grade inflation. RESULTS: Meta-regression analyzed 57 separate predictor-outcome correlations (POCs) and construct-level predictive validities (CLPVs). Mean CLPVs are substantially higher (.450) than mean POCs (.171). Mean CLPVs for first-year examinations, were high for A-levels (.809; CI: .501 to .935), and lower for GCSEs/O-levels (.332; CI: .024 to .583) and UKCAT (mean = .245; CI: .207 to .276). A-levels had higher CLPVs for all undergraduate and postgraduate assessments than did GCSEs/O-levels and intellectual aptitude tests. CLPVs of educational attainment measures decline somewhat during training, but continue to predict postgraduate performance. Intellectual aptitude tests have lower CLPVs than A-levels or GCSEs/O-levels. CONCLUSIONS: Educational attainment has strong CLPVs for undergraduate and postgraduate performance, accounting for perhaps 65% of true variance in first year performance. Such CLPVs justify the use of educational attainment measure in selection, but also raise a key theoretical question concerning the remaining 35% of variance (and measurement error, range restriction and right-censorship have been taken into account). Just as in astrophysics, ‘dark matter’ and ‘dark energy’ are posited to balance various theoretical equations, so medical student selection must also have its ‘dark variance’, whose nature is not yet properly characterized, but explains a third of the variation in performance during training. Some variance probably relates to factors which are unpredictable at selection, such as illness or other life events, but some is probably also associated with factors such as personality, motivation or study skills.
format Online
Article
Text
id pubmed-3827328
institution National Center for Biotechnology Information
language English
publishDate 2013
publisher BioMed Central
record_format MEDLINE/PubMed
spelling pubmed-38273282013-11-18 Construct-level predictive validity of educational attainment and intellectual aptitude tests in medical student selection: meta-regression of six UK longitudinal studies McManus, IC Dewberry, Chris Nicholson, Sandra Dowell, Jonathan S Woolf, Katherine Potts, Henry WW BMC Med Research Article BACKGROUND: Measures used for medical student selection should predict future performance during training. A problem for any selection study is that predictor-outcome correlations are known only in those who have been selected, whereas selectors need to know how measures would predict in the entire pool of applicants. That problem of interpretation can be solved by calculating construct-level predictive validity, an estimate of true predictor-outcome correlation across the range of applicant abilities. METHODS: Construct-level predictive validities were calculated in six cohort studies of medical student selection and training (student entry, 1972 to 2009) for a range of predictors, including A-levels, General Certificates of Secondary Education (GCSEs)/O-levels, and aptitude tests (AH5 and UK Clinical Aptitude Test (UKCAT)). Outcomes included undergraduate basic medical science and finals assessments, as well as postgraduate measures of Membership of the Royal Colleges of Physicians of the United Kingdom (MRCP(UK)) performance and entry in the Specialist Register. Construct-level predictive validity was calculated with the method of Hunter, Schmidt and Le (2006), adapted to correct for right-censorship of examination results due to grade inflation. RESULTS: Meta-regression analyzed 57 separate predictor-outcome correlations (POCs) and construct-level predictive validities (CLPVs). Mean CLPVs are substantially higher (.450) than mean POCs (.171). Mean CLPVs for first-year examinations, were high for A-levels (.809; CI: .501 to .935), and lower for GCSEs/O-levels (.332; CI: .024 to .583) and UKCAT (mean = .245; CI: .207 to .276). A-levels had higher CLPVs for all undergraduate and postgraduate assessments than did GCSEs/O-levels and intellectual aptitude tests. CLPVs of educational attainment measures decline somewhat during training, but continue to predict postgraduate performance. Intellectual aptitude tests have lower CLPVs than A-levels or GCSEs/O-levels. CONCLUSIONS: Educational attainment has strong CLPVs for undergraduate and postgraduate performance, accounting for perhaps 65% of true variance in first year performance. Such CLPVs justify the use of educational attainment measure in selection, but also raise a key theoretical question concerning the remaining 35% of variance (and measurement error, range restriction and right-censorship have been taken into account). Just as in astrophysics, ‘dark matter’ and ‘dark energy’ are posited to balance various theoretical equations, so medical student selection must also have its ‘dark variance’, whose nature is not yet properly characterized, but explains a third of the variation in performance during training. Some variance probably relates to factors which are unpredictable at selection, such as illness or other life events, but some is probably also associated with factors such as personality, motivation or study skills. BioMed Central 2013-11-14 /pmc/articles/PMC3827328/ /pubmed/24229353 http://dx.doi.org/10.1186/1741-7015-11-243 Text en Copyright © 2013 McManus et al.; licensee BioMed Central Ltd. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
spellingShingle Research Article
McManus, IC
Dewberry, Chris
Nicholson, Sandra
Dowell, Jonathan S
Woolf, Katherine
Potts, Henry WW
Construct-level predictive validity of educational attainment and intellectual aptitude tests in medical student selection: meta-regression of six UK longitudinal studies
title Construct-level predictive validity of educational attainment and intellectual aptitude tests in medical student selection: meta-regression of six UK longitudinal studies
title_full Construct-level predictive validity of educational attainment and intellectual aptitude tests in medical student selection: meta-regression of six UK longitudinal studies
title_fullStr Construct-level predictive validity of educational attainment and intellectual aptitude tests in medical student selection: meta-regression of six UK longitudinal studies
title_full_unstemmed Construct-level predictive validity of educational attainment and intellectual aptitude tests in medical student selection: meta-regression of six UK longitudinal studies
title_short Construct-level predictive validity of educational attainment and intellectual aptitude tests in medical student selection: meta-regression of six UK longitudinal studies
title_sort construct-level predictive validity of educational attainment and intellectual aptitude tests in medical student selection: meta-regression of six uk longitudinal studies
topic Research Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3827328/
https://www.ncbi.nlm.nih.gov/pubmed/24229353
http://dx.doi.org/10.1186/1741-7015-11-243
work_keys_str_mv AT mcmanusic constructlevelpredictivevalidityofeducationalattainmentandintellectualaptitudetestsinmedicalstudentselectionmetaregressionofsixuklongitudinalstudies
AT dewberrychris constructlevelpredictivevalidityofeducationalattainmentandintellectualaptitudetestsinmedicalstudentselectionmetaregressionofsixuklongitudinalstudies
AT nicholsonsandra constructlevelpredictivevalidityofeducationalattainmentandintellectualaptitudetestsinmedicalstudentselectionmetaregressionofsixuklongitudinalstudies
AT dowelljonathans constructlevelpredictivevalidityofeducationalattainmentandintellectualaptitudetestsinmedicalstudentselectionmetaregressionofsixuklongitudinalstudies
AT woolfkatherine constructlevelpredictivevalidityofeducationalattainmentandintellectualaptitudetestsinmedicalstudentselectionmetaregressionofsixuklongitudinalstudies
AT pottshenryww constructlevelpredictivevalidityofeducationalattainmentandintellectualaptitudetestsinmedicalstudentselectionmetaregressionofsixuklongitudinalstudies