Cargando…

Using Multiverse Analysis to Highlight Differences in Convergent Correlation Outcomes Due to Data Analytical and Study Design Choices

In neuropsychological research, there are a near limitless number of different approaches researchers can choose when designing studies. Here we showcase the multiverse/specification curve technique to establish the robustness of analytical pathways choices within classic psychometric test validatio...

Descripción completa

Detalles Bibliográficos
Autores principales: Webb, Sam S., Demeyere, Nele
Formato: Online Artículo Texto
Lenguaje:English
Publicado: SAGE Publications 2022
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10363922/
https://www.ncbi.nlm.nih.gov/pubmed/36176188
http://dx.doi.org/10.1177/10731911221127904
_version_ 1785076741559025664
author Webb, Sam S.
Demeyere, Nele
author_facet Webb, Sam S.
Demeyere, Nele
author_sort Webb, Sam S.
collection PubMed
description In neuropsychological research, there are a near limitless number of different approaches researchers can choose when designing studies. Here we showcase the multiverse/specification curve technique to establish the robustness of analytical pathways choices within classic psychometric test validation in an example test of executive function. We examined the impact of choices regarding sample groups, sample sizes, test metrics, and covariate inclusions on convergent validation correlations between tests of executive function. Data were available for 87 neurologically healthy adults and 117 stroke survivors, and a total of 2,220 different analyses were run in a multiverse analysis. We found that the type of sample group, sample size, and test metric used for analyses affected validation outcomes. Covariate inclusion choices did not affect the observed coefficients in our analyses. The present analysis demonstrates the importance of carefully justifying every aspect of a psychometric test validation study a priori with theoretical and statistical factors in mind. It is essential to thoroughly consider the purpose and use of a new tool when designing validation studies.
format Online
Article
Text
id pubmed-10363922
institution National Center for Biotechnology Information
language English
publishDate 2022
publisher SAGE Publications
record_format MEDLINE/PubMed
spelling pubmed-103639222023-07-25 Using Multiverse Analysis to Highlight Differences in Convergent Correlation Outcomes Due to Data Analytical and Study Design Choices Webb, Sam S. Demeyere, Nele Assessment Articles In neuropsychological research, there are a near limitless number of different approaches researchers can choose when designing studies. Here we showcase the multiverse/specification curve technique to establish the robustness of analytical pathways choices within classic psychometric test validation in an example test of executive function. We examined the impact of choices regarding sample groups, sample sizes, test metrics, and covariate inclusions on convergent validation correlations between tests of executive function. Data were available for 87 neurologically healthy adults and 117 stroke survivors, and a total of 2,220 different analyses were run in a multiverse analysis. We found that the type of sample group, sample size, and test metric used for analyses affected validation outcomes. Covariate inclusion choices did not affect the observed coefficients in our analyses. The present analysis demonstrates the importance of carefully justifying every aspect of a psychometric test validation study a priori with theoretical and statistical factors in mind. It is essential to thoroughly consider the purpose and use of a new tool when designing validation studies. SAGE Publications 2022-09-29 2023-09 /pmc/articles/PMC10363922/ /pubmed/36176188 http://dx.doi.org/10.1177/10731911221127904 Text en © The Author(s) 2022 https://creativecommons.org/licenses/by/4.0/This article is distributed under the terms of the Creative Commons Attribution 4.0 License (https://creativecommons.org/licenses/by/4.0/) which permits any use, reproduction and distribution of the work without further permission provided the original work is attributed as specified on the SAGE and Open Access page (https://us.sagepub.com/en-us/nam/open-access-at-sage).
spellingShingle Articles
Webb, Sam S.
Demeyere, Nele
Using Multiverse Analysis to Highlight Differences in Convergent Correlation Outcomes Due to Data Analytical and Study Design Choices
title Using Multiverse Analysis to Highlight Differences in Convergent Correlation Outcomes Due to Data Analytical and Study Design Choices
title_full Using Multiverse Analysis to Highlight Differences in Convergent Correlation Outcomes Due to Data Analytical and Study Design Choices
title_fullStr Using Multiverse Analysis to Highlight Differences in Convergent Correlation Outcomes Due to Data Analytical and Study Design Choices
title_full_unstemmed Using Multiverse Analysis to Highlight Differences in Convergent Correlation Outcomes Due to Data Analytical and Study Design Choices
title_short Using Multiverse Analysis to Highlight Differences in Convergent Correlation Outcomes Due to Data Analytical and Study Design Choices
title_sort using multiverse analysis to highlight differences in convergent correlation outcomes due to data analytical and study design choices
topic Articles
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10363922/
https://www.ncbi.nlm.nih.gov/pubmed/36176188
http://dx.doi.org/10.1177/10731911221127904
work_keys_str_mv AT webbsams usingmultiverseanalysistohighlightdifferencesinconvergentcorrelationoutcomesduetodataanalyticalandstudydesignchoices
AT demeyerenele usingmultiverseanalysistohighlightdifferencesinconvergentcorrelationoutcomesduetodataanalyticalandstudydesignchoices