Cargando…

In models we trust: preregistration, large samples, and replication may not suffice

Despite discussions about the replicability of findings in psychological research, two issues have been largely ignored: selection mechanisms and model assumptions. Both topics address the same fundamental question: Does the chosen statistical analysis tool adequately model the data generation proce...

Descripción completa

Detalles Bibliográficos
Autores principales: Spiess, Martin, Jordan, Pascal
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Frontiers Media S.A. 2023
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10551181/
https://www.ncbi.nlm.nih.gov/pubmed/37809287
http://dx.doi.org/10.3389/fpsyg.2023.1266447
_version_ 1785115706338050048
author Spiess, Martin
Jordan, Pascal
author_facet Spiess, Martin
Jordan, Pascal
author_sort Spiess, Martin
collection PubMed
description Despite discussions about the replicability of findings in psychological research, two issues have been largely ignored: selection mechanisms and model assumptions. Both topics address the same fundamental question: Does the chosen statistical analysis tool adequately model the data generation process? In this article, we address both issues and show, in a first step, that in the face of selective samples and contrary to common practice, the validity of inferences, even when based on experimental designs, can be claimed without further justification and adaptation of standard methods only in very specific situations. We then broaden our perspective to discuss consequences of violated assumptions in linear models in the context of psychological research in general and in generalized linear mixed models as used in item response theory. These types of misspecification are oftentimes ignored in the psychological research literature. It is emphasized that the above problems cannot be overcome by strategies such as preregistration, large samples, replications, or a ban on testing null hypotheses. To avoid biased conclusions, we briefly discuss tools such as model diagnostics, statistical methods to compensate for selectivity and semi- or non-parametric estimation. At a more fundamental level, however, a twofold strategy seems indispensable: (1) iterative, cumulative theory development based on statistical methods with theoretically justified assumptions, and (2) empirical research on variables that affect (self-) selection into the observed part of the sample and the use of this information to compensate for selectivity.
format Online
Article
Text
id pubmed-10551181
institution National Center for Biotechnology Information
language English
publishDate 2023
publisher Frontiers Media S.A.
record_format MEDLINE/PubMed
spelling pubmed-105511812023-10-06 In models we trust: preregistration, large samples, and replication may not suffice Spiess, Martin Jordan, Pascal Front Psychol Psychology Despite discussions about the replicability of findings in psychological research, two issues have been largely ignored: selection mechanisms and model assumptions. Both topics address the same fundamental question: Does the chosen statistical analysis tool adequately model the data generation process? In this article, we address both issues and show, in a first step, that in the face of selective samples and contrary to common practice, the validity of inferences, even when based on experimental designs, can be claimed without further justification and adaptation of standard methods only in very specific situations. We then broaden our perspective to discuss consequences of violated assumptions in linear models in the context of psychological research in general and in generalized linear mixed models as used in item response theory. These types of misspecification are oftentimes ignored in the psychological research literature. It is emphasized that the above problems cannot be overcome by strategies such as preregistration, large samples, replications, or a ban on testing null hypotheses. To avoid biased conclusions, we briefly discuss tools such as model diagnostics, statistical methods to compensate for selectivity and semi- or non-parametric estimation. At a more fundamental level, however, a twofold strategy seems indispensable: (1) iterative, cumulative theory development based on statistical methods with theoretically justified assumptions, and (2) empirical research on variables that affect (self-) selection into the observed part of the sample and the use of this information to compensate for selectivity. Frontiers Media S.A. 2023-09-21 /pmc/articles/PMC10551181/ /pubmed/37809287 http://dx.doi.org/10.3389/fpsyg.2023.1266447 Text en Copyright © 2023 Spiess and Jordan. https://creativecommons.org/licenses/by/4.0/This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
spellingShingle Psychology
Spiess, Martin
Jordan, Pascal
In models we trust: preregistration, large samples, and replication may not suffice
title In models we trust: preregistration, large samples, and replication may not suffice
title_full In models we trust: preregistration, large samples, and replication may not suffice
title_fullStr In models we trust: preregistration, large samples, and replication may not suffice
title_full_unstemmed In models we trust: preregistration, large samples, and replication may not suffice
title_short In models we trust: preregistration, large samples, and replication may not suffice
title_sort in models we trust: preregistration, large samples, and replication may not suffice
topic Psychology
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10551181/
https://www.ncbi.nlm.nih.gov/pubmed/37809287
http://dx.doi.org/10.3389/fpsyg.2023.1266447
work_keys_str_mv AT spiessmartin inmodelswetrustpreregistrationlargesamplesandreplicationmaynotsuffice
AT jordanpascal inmodelswetrustpreregistrationlargesamplesandreplicationmaynotsuffice