Cargando…
A ’compensatory selection’ effect with standardized tests: Lack of correlation between test scores and success is evidence that test scores are predictive of success
We introduce the statistical concept of ’compensatory selection’, which arises when selecting a subset of applicants based on multiple predictors, such as when standardized test scores are used in combination with other predictors required in a school application (e.g., previous grades, references l...
Autores principales: | , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Public Library of Science
2022
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9119552/ https://www.ncbi.nlm.nih.gov/pubmed/35588112 http://dx.doi.org/10.1371/journal.pone.0265459 |
_version_ | 1784710726802210816 |
---|---|
author | Huber, David E. Cohen, Andrew L. Staub, Adrian |
author_facet | Huber, David E. Cohen, Andrew L. Staub, Adrian |
author_sort | Huber, David E. |
collection | PubMed |
description | We introduce the statistical concept of ’compensatory selection’, which arises when selecting a subset of applicants based on multiple predictors, such as when standardized test scores are used in combination with other predictors required in a school application (e.g., previous grades, references letters, and personal statements). Post-hoc analyses often fail to find a positive correlation between test scores and subsequent success, and this failure is sometimes taken as evidence against the predictive validity of the standardized test. The present analysis reveals that the failure to find a negative correlation indicates that the standardized test is in fact a valid predictor of success. This is due to compensation between predictors during selection: Some students are admitted despite a low test score because their application is exceptional in other respects, while other students are admitted primarily based on a high test score despite weakness in the rest of their application. This compensatory selection process introduces a negative correlation between test scores and other predictors among those admitted (a ’collider bias’ or ’Berkson’s paradox’ effect). If test scores are valid predictors of success, this negative correlation between the predictors counteracts the positive correlation between test scores and success that would have been observed if all applicants were admitted. If test scores are not predictive of success, but were nevertheless used in a compensatory selection process, there would be a spurious negative correlation between test scores and success (i.e., an admitted student with a weak application except for a high test score would be unlikely to succeed). The selection effect that is described here is fundamentally different from the well-known ’restricted range’ problem and can powerfully alter results even in situations that accept most applicants. |
format | Online Article Text |
id | pubmed-9119552 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2022 |
publisher | Public Library of Science |
record_format | MEDLINE/PubMed |
spelling | pubmed-91195522022-05-20 A ’compensatory selection’ effect with standardized tests: Lack of correlation between test scores and success is evidence that test scores are predictive of success Huber, David E. Cohen, Andrew L. Staub, Adrian PLoS One Research Article We introduce the statistical concept of ’compensatory selection’, which arises when selecting a subset of applicants based on multiple predictors, such as when standardized test scores are used in combination with other predictors required in a school application (e.g., previous grades, references letters, and personal statements). Post-hoc analyses often fail to find a positive correlation between test scores and subsequent success, and this failure is sometimes taken as evidence against the predictive validity of the standardized test. The present analysis reveals that the failure to find a negative correlation indicates that the standardized test is in fact a valid predictor of success. This is due to compensation between predictors during selection: Some students are admitted despite a low test score because their application is exceptional in other respects, while other students are admitted primarily based on a high test score despite weakness in the rest of their application. This compensatory selection process introduces a negative correlation between test scores and other predictors among those admitted (a ’collider bias’ or ’Berkson’s paradox’ effect). If test scores are valid predictors of success, this negative correlation between the predictors counteracts the positive correlation between test scores and success that would have been observed if all applicants were admitted. If test scores are not predictive of success, but were nevertheless used in a compensatory selection process, there would be a spurious negative correlation between test scores and success (i.e., an admitted student with a weak application except for a high test score would be unlikely to succeed). The selection effect that is described here is fundamentally different from the well-known ’restricted range’ problem and can powerfully alter results even in situations that accept most applicants. Public Library of Science 2022-05-19 /pmc/articles/PMC9119552/ /pubmed/35588112 http://dx.doi.org/10.1371/journal.pone.0265459 Text en © 2022 Huber et al https://creativecommons.org/licenses/by/4.0/This is an open access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/) , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. |
spellingShingle | Research Article Huber, David E. Cohen, Andrew L. Staub, Adrian A ’compensatory selection’ effect with standardized tests: Lack of correlation between test scores and success is evidence that test scores are predictive of success |
title | A ’compensatory selection’ effect with standardized tests: Lack of correlation between test scores and success is evidence that test scores are predictive of success |
title_full | A ’compensatory selection’ effect with standardized tests: Lack of correlation between test scores and success is evidence that test scores are predictive of success |
title_fullStr | A ’compensatory selection’ effect with standardized tests: Lack of correlation between test scores and success is evidence that test scores are predictive of success |
title_full_unstemmed | A ’compensatory selection’ effect with standardized tests: Lack of correlation between test scores and success is evidence that test scores are predictive of success |
title_short | A ’compensatory selection’ effect with standardized tests: Lack of correlation between test scores and success is evidence that test scores are predictive of success |
title_sort | ’compensatory selection’ effect with standardized tests: lack of correlation between test scores and success is evidence that test scores are predictive of success |
topic | Research Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9119552/ https://www.ncbi.nlm.nih.gov/pubmed/35588112 http://dx.doi.org/10.1371/journal.pone.0265459 |
work_keys_str_mv | AT huberdavide acompensatoryselectioneffectwithstandardizedtestslackofcorrelationbetweentestscoresandsuccessisevidencethattestscoresarepredictiveofsuccess AT cohenandrewl acompensatoryselectioneffectwithstandardizedtestslackofcorrelationbetweentestscoresandsuccessisevidencethattestscoresarepredictiveofsuccess AT staubadrian acompensatoryselectioneffectwithstandardizedtestslackofcorrelationbetweentestscoresandsuccessisevidencethattestscoresarepredictiveofsuccess AT huberdavide compensatoryselectioneffectwithstandardizedtestslackofcorrelationbetweentestscoresandsuccessisevidencethattestscoresarepredictiveofsuccess AT cohenandrewl compensatoryselectioneffectwithstandardizedtestslackofcorrelationbetweentestscoresandsuccessisevidencethattestscoresarepredictiveofsuccess AT staubadrian compensatoryselectioneffectwithstandardizedtestslackofcorrelationbetweentestscoresandsuccessisevidencethattestscoresarepredictiveofsuccess |