Cargando…
Statistically Controlling for Confounding Constructs Is Harder than You Think
Social scientists often seek to demonstrate that a construct has incremental validity over and above other related constructs. However, these claims are typically supported by measurement-level models that fail to consider the effects of measurement (un)reliability. We use intuitive examples, Monte...
Autores principales: | , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Public Library of Science
2016
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4816570/ https://www.ncbi.nlm.nih.gov/pubmed/27031707 http://dx.doi.org/10.1371/journal.pone.0152719 |
_version_ | 1782424741603180544 |
---|---|
author | Westfall, Jacob Yarkoni, Tal |
author_facet | Westfall, Jacob Yarkoni, Tal |
author_sort | Westfall, Jacob |
collection | PubMed |
description | Social scientists often seek to demonstrate that a construct has incremental validity over and above other related constructs. However, these claims are typically supported by measurement-level models that fail to consider the effects of measurement (un)reliability. We use intuitive examples, Monte Carlo simulations, and a novel analytical framework to demonstrate that common strategies for establishing incremental construct validity using multiple regression analysis exhibit extremely high Type I error rates under parameter regimes common in many psychological domains. Counterintuitively, we find that error rates are highest—in some cases approaching 100%—when sample sizes are large and reliability is moderate. Our findings suggest that a potentially large proportion of incremental validity claims made in the literature are spurious. We present a web application (http://jakewestfall.org/ivy/) that readers can use to explore the statistical properties of these and other incremental validity arguments. We conclude by reviewing SEM-based statistical approaches that appropriately control the Type I error rate when attempting to establish incremental validity. |
format | Online Article Text |
id | pubmed-4816570 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2016 |
publisher | Public Library of Science |
record_format | MEDLINE/PubMed |
spelling | pubmed-48165702016-04-14 Statistically Controlling for Confounding Constructs Is Harder than You Think Westfall, Jacob Yarkoni, Tal PLoS One Research Article Social scientists often seek to demonstrate that a construct has incremental validity over and above other related constructs. However, these claims are typically supported by measurement-level models that fail to consider the effects of measurement (un)reliability. We use intuitive examples, Monte Carlo simulations, and a novel analytical framework to demonstrate that common strategies for establishing incremental construct validity using multiple regression analysis exhibit extremely high Type I error rates under parameter regimes common in many psychological domains. Counterintuitively, we find that error rates are highest—in some cases approaching 100%—when sample sizes are large and reliability is moderate. Our findings suggest that a potentially large proportion of incremental validity claims made in the literature are spurious. We present a web application (http://jakewestfall.org/ivy/) that readers can use to explore the statistical properties of these and other incremental validity arguments. We conclude by reviewing SEM-based statistical approaches that appropriately control the Type I error rate when attempting to establish incremental validity. Public Library of Science 2016-03-31 /pmc/articles/PMC4816570/ /pubmed/27031707 http://dx.doi.org/10.1371/journal.pone.0152719 Text en © 2016 Westfall, Yarkoni http://creativecommons.org/licenses/by/4.0/ This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/) , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. |
spellingShingle | Research Article Westfall, Jacob Yarkoni, Tal Statistically Controlling for Confounding Constructs Is Harder than You Think |
title | Statistically Controlling for Confounding Constructs Is Harder than You Think |
title_full | Statistically Controlling for Confounding Constructs Is Harder than You Think |
title_fullStr | Statistically Controlling for Confounding Constructs Is Harder than You Think |
title_full_unstemmed | Statistically Controlling for Confounding Constructs Is Harder than You Think |
title_short | Statistically Controlling for Confounding Constructs Is Harder than You Think |
title_sort | statistically controlling for confounding constructs is harder than you think |
topic | Research Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4816570/ https://www.ncbi.nlm.nih.gov/pubmed/27031707 http://dx.doi.org/10.1371/journal.pone.0152719 |
work_keys_str_mv | AT westfalljacob statisticallycontrollingforconfoundingconstructsisharderthanyouthink AT yarkonital statisticallycontrollingforconfoundingconstructsisharderthanyouthink |