Cargando…
In models we trust: preregistration, large samples, and replication may not suffice
Despite discussions about the replicability of findings in psychological research, two issues have been largely ignored: selection mechanisms and model assumptions. Both topics address the same fundamental question: Does the chosen statistical analysis tool adequately model the data generation proce...
Autores principales: | , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Frontiers Media S.A.
2023
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10551181/ https://www.ncbi.nlm.nih.gov/pubmed/37809287 http://dx.doi.org/10.3389/fpsyg.2023.1266447 |
_version_ | 1785115706338050048 |
---|---|
author | Spiess, Martin Jordan, Pascal |
author_facet | Spiess, Martin Jordan, Pascal |
author_sort | Spiess, Martin |
collection | PubMed |
description | Despite discussions about the replicability of findings in psychological research, two issues have been largely ignored: selection mechanisms and model assumptions. Both topics address the same fundamental question: Does the chosen statistical analysis tool adequately model the data generation process? In this article, we address both issues and show, in a first step, that in the face of selective samples and contrary to common practice, the validity of inferences, even when based on experimental designs, can be claimed without further justification and adaptation of standard methods only in very specific situations. We then broaden our perspective to discuss consequences of violated assumptions in linear models in the context of psychological research in general and in generalized linear mixed models as used in item response theory. These types of misspecification are oftentimes ignored in the psychological research literature. It is emphasized that the above problems cannot be overcome by strategies such as preregistration, large samples, replications, or a ban on testing null hypotheses. To avoid biased conclusions, we briefly discuss tools such as model diagnostics, statistical methods to compensate for selectivity and semi- or non-parametric estimation. At a more fundamental level, however, a twofold strategy seems indispensable: (1) iterative, cumulative theory development based on statistical methods with theoretically justified assumptions, and (2) empirical research on variables that affect (self-) selection into the observed part of the sample and the use of this information to compensate for selectivity. |
format | Online Article Text |
id | pubmed-10551181 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2023 |
publisher | Frontiers Media S.A. |
record_format | MEDLINE/PubMed |
spelling | pubmed-105511812023-10-06 In models we trust: preregistration, large samples, and replication may not suffice Spiess, Martin Jordan, Pascal Front Psychol Psychology Despite discussions about the replicability of findings in psychological research, two issues have been largely ignored: selection mechanisms and model assumptions. Both topics address the same fundamental question: Does the chosen statistical analysis tool adequately model the data generation process? In this article, we address both issues and show, in a first step, that in the face of selective samples and contrary to common practice, the validity of inferences, even when based on experimental designs, can be claimed without further justification and adaptation of standard methods only in very specific situations. We then broaden our perspective to discuss consequences of violated assumptions in linear models in the context of psychological research in general and in generalized linear mixed models as used in item response theory. These types of misspecification are oftentimes ignored in the psychological research literature. It is emphasized that the above problems cannot be overcome by strategies such as preregistration, large samples, replications, or a ban on testing null hypotheses. To avoid biased conclusions, we briefly discuss tools such as model diagnostics, statistical methods to compensate for selectivity and semi- or non-parametric estimation. At a more fundamental level, however, a twofold strategy seems indispensable: (1) iterative, cumulative theory development based on statistical methods with theoretically justified assumptions, and (2) empirical research on variables that affect (self-) selection into the observed part of the sample and the use of this information to compensate for selectivity. Frontiers Media S.A. 2023-09-21 /pmc/articles/PMC10551181/ /pubmed/37809287 http://dx.doi.org/10.3389/fpsyg.2023.1266447 Text en Copyright © 2023 Spiess and Jordan. https://creativecommons.org/licenses/by/4.0/This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms. |
spellingShingle | Psychology Spiess, Martin Jordan, Pascal In models we trust: preregistration, large samples, and replication may not suffice |
title | In models we trust: preregistration, large samples, and replication may not suffice |
title_full | In models we trust: preregistration, large samples, and replication may not suffice |
title_fullStr | In models we trust: preregistration, large samples, and replication may not suffice |
title_full_unstemmed | In models we trust: preregistration, large samples, and replication may not suffice |
title_short | In models we trust: preregistration, large samples, and replication may not suffice |
title_sort | in models we trust: preregistration, large samples, and replication may not suffice |
topic | Psychology |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10551181/ https://www.ncbi.nlm.nih.gov/pubmed/37809287 http://dx.doi.org/10.3389/fpsyg.2023.1266447 |
work_keys_str_mv | AT spiessmartin inmodelswetrustpreregistrationlargesamplesandreplicationmaynotsuffice AT jordanpascal inmodelswetrustpreregistrationlargesamplesandreplicationmaynotsuffice |