Cargando…

A genetic algorithm for optimal assembly of pairwise forced-choice questionnaires

The use of multidimensional forced-choice questionnaires has been proposed as a means of improving validity in the assessment of non-cognitive attributes in high-stakes scenarios. However, the reduced precision of trait estimates in this questionnaire format is an important drawback. Accordingly, th...

Descripción completa

Detalles Bibliográficos
Autores principales: Kreitchmann, Rodrigo Schames, Abad, Francisco J., Sorrel, Miguel A.
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Springer US 2021
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9170671/
https://www.ncbi.nlm.nih.gov/pubmed/34505277
http://dx.doi.org/10.3758/s13428-021-01677-4
_version_ 1784721486601256960
author Kreitchmann, Rodrigo Schames
Abad, Francisco J.
Sorrel, Miguel A.
author_facet Kreitchmann, Rodrigo Schames
Abad, Francisco J.
Sorrel, Miguel A.
author_sort Kreitchmann, Rodrigo Schames
collection PubMed
description The use of multidimensional forced-choice questionnaires has been proposed as a means of improving validity in the assessment of non-cognitive attributes in high-stakes scenarios. However, the reduced precision of trait estimates in this questionnaire format is an important drawback. Accordingly, this article presents an optimization procedure for assembling pairwise forced-choice questionnaires while maximizing posterior marginal reliabilities. This procedure is performed through the adaptation of a known genetic algorithm (GA) for combinatorial problems. In a simulation study, the efficiency of the proposed procedure was compared with a quasi-brute-force (BF) search. For this purpose, five-dimensional item pools were simulated to emulate the real problem of generating a forced-choice personality questionnaire under the five-factor model. Three factors were manipulated: (1) the length of the questionnaire, (2) the relative item pool size with respect to the questionnaire’s length, and (3) the true correlations between traits. The recovery of the person parameters for each assembled questionnaire was evaluated through the squared correlation between estimated and true parameters, the root mean square error between the estimated and true parameters, the average difference between the estimated and true inter-trait correlations, and the average standard error for each trait level. The proposed GA offered more accurate trait estimates than the BF search within a reasonable computation time in every simulation condition. Such improvements were especially important when measuring correlated traits and when the relative item pool sizes were higher. A user-friendly online implementation of the algorithm was made available to the users.
format Online
Article
Text
id pubmed-9170671
institution National Center for Biotechnology Information
language English
publishDate 2021
publisher Springer US
record_format MEDLINE/PubMed
spelling pubmed-91706712022-06-08 A genetic algorithm for optimal assembly of pairwise forced-choice questionnaires Kreitchmann, Rodrigo Schames Abad, Francisco J. Sorrel, Miguel A. Behav Res Methods Article The use of multidimensional forced-choice questionnaires has been proposed as a means of improving validity in the assessment of non-cognitive attributes in high-stakes scenarios. However, the reduced precision of trait estimates in this questionnaire format is an important drawback. Accordingly, this article presents an optimization procedure for assembling pairwise forced-choice questionnaires while maximizing posterior marginal reliabilities. This procedure is performed through the adaptation of a known genetic algorithm (GA) for combinatorial problems. In a simulation study, the efficiency of the proposed procedure was compared with a quasi-brute-force (BF) search. For this purpose, five-dimensional item pools were simulated to emulate the real problem of generating a forced-choice personality questionnaire under the five-factor model. Three factors were manipulated: (1) the length of the questionnaire, (2) the relative item pool size with respect to the questionnaire’s length, and (3) the true correlations between traits. The recovery of the person parameters for each assembled questionnaire was evaluated through the squared correlation between estimated and true parameters, the root mean square error between the estimated and true parameters, the average difference between the estimated and true inter-trait correlations, and the average standard error for each trait level. The proposed GA offered more accurate trait estimates than the BF search within a reasonable computation time in every simulation condition. Such improvements were especially important when measuring correlated traits and when the relative item pool sizes were higher. A user-friendly online implementation of the algorithm was made available to the users. Springer US 2021-09-09 2022 /pmc/articles/PMC9170671/ /pubmed/34505277 http://dx.doi.org/10.3758/s13428-021-01677-4 Text en © The Author(s) 2021 https://creativecommons.org/licenses/by/4.0/Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ (https://creativecommons.org/licenses/by/4.0/) .
spellingShingle Article
Kreitchmann, Rodrigo Schames
Abad, Francisco J.
Sorrel, Miguel A.
A genetic algorithm for optimal assembly of pairwise forced-choice questionnaires
title A genetic algorithm for optimal assembly of pairwise forced-choice questionnaires
title_full A genetic algorithm for optimal assembly of pairwise forced-choice questionnaires
title_fullStr A genetic algorithm for optimal assembly of pairwise forced-choice questionnaires
title_full_unstemmed A genetic algorithm for optimal assembly of pairwise forced-choice questionnaires
title_short A genetic algorithm for optimal assembly of pairwise forced-choice questionnaires
title_sort genetic algorithm for optimal assembly of pairwise forced-choice questionnaires
topic Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9170671/
https://www.ncbi.nlm.nih.gov/pubmed/34505277
http://dx.doi.org/10.3758/s13428-021-01677-4
work_keys_str_mv AT kreitchmannrodrigoschames ageneticalgorithmforoptimalassemblyofpairwiseforcedchoicequestionnaires
AT abadfranciscoj ageneticalgorithmforoptimalassemblyofpairwiseforcedchoicequestionnaires
AT sorrelmiguela ageneticalgorithmforoptimalassemblyofpairwiseforcedchoicequestionnaires
AT kreitchmannrodrigoschames geneticalgorithmforoptimalassemblyofpairwiseforcedchoicequestionnaires
AT abadfranciscoj geneticalgorithmforoptimalassemblyofpairwiseforcedchoicequestionnaires
AT sorrelmiguela geneticalgorithmforoptimalassemblyofpairwiseforcedchoicequestionnaires