Cargando…
A genetic algorithm for optimal assembly of pairwise forced-choice questionnaires
The use of multidimensional forced-choice questionnaires has been proposed as a means of improving validity in the assessment of non-cognitive attributes in high-stakes scenarios. However, the reduced precision of trait estimates in this questionnaire format is an important drawback. Accordingly, th...
Autores principales: | , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Springer US
2021
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9170671/ https://www.ncbi.nlm.nih.gov/pubmed/34505277 http://dx.doi.org/10.3758/s13428-021-01677-4 |
_version_ | 1784721486601256960 |
---|---|
author | Kreitchmann, Rodrigo Schames Abad, Francisco J. Sorrel, Miguel A. |
author_facet | Kreitchmann, Rodrigo Schames Abad, Francisco J. Sorrel, Miguel A. |
author_sort | Kreitchmann, Rodrigo Schames |
collection | PubMed |
description | The use of multidimensional forced-choice questionnaires has been proposed as a means of improving validity in the assessment of non-cognitive attributes in high-stakes scenarios. However, the reduced precision of trait estimates in this questionnaire format is an important drawback. Accordingly, this article presents an optimization procedure for assembling pairwise forced-choice questionnaires while maximizing posterior marginal reliabilities. This procedure is performed through the adaptation of a known genetic algorithm (GA) for combinatorial problems. In a simulation study, the efficiency of the proposed procedure was compared with a quasi-brute-force (BF) search. For this purpose, five-dimensional item pools were simulated to emulate the real problem of generating a forced-choice personality questionnaire under the five-factor model. Three factors were manipulated: (1) the length of the questionnaire, (2) the relative item pool size with respect to the questionnaire’s length, and (3) the true correlations between traits. The recovery of the person parameters for each assembled questionnaire was evaluated through the squared correlation between estimated and true parameters, the root mean square error between the estimated and true parameters, the average difference between the estimated and true inter-trait correlations, and the average standard error for each trait level. The proposed GA offered more accurate trait estimates than the BF search within a reasonable computation time in every simulation condition. Such improvements were especially important when measuring correlated traits and when the relative item pool sizes were higher. A user-friendly online implementation of the algorithm was made available to the users. |
format | Online Article Text |
id | pubmed-9170671 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2021 |
publisher | Springer US |
record_format | MEDLINE/PubMed |
spelling | pubmed-91706712022-06-08 A genetic algorithm for optimal assembly of pairwise forced-choice questionnaires Kreitchmann, Rodrigo Schames Abad, Francisco J. Sorrel, Miguel A. Behav Res Methods Article The use of multidimensional forced-choice questionnaires has been proposed as a means of improving validity in the assessment of non-cognitive attributes in high-stakes scenarios. However, the reduced precision of trait estimates in this questionnaire format is an important drawback. Accordingly, this article presents an optimization procedure for assembling pairwise forced-choice questionnaires while maximizing posterior marginal reliabilities. This procedure is performed through the adaptation of a known genetic algorithm (GA) for combinatorial problems. In a simulation study, the efficiency of the proposed procedure was compared with a quasi-brute-force (BF) search. For this purpose, five-dimensional item pools were simulated to emulate the real problem of generating a forced-choice personality questionnaire under the five-factor model. Three factors were manipulated: (1) the length of the questionnaire, (2) the relative item pool size with respect to the questionnaire’s length, and (3) the true correlations between traits. The recovery of the person parameters for each assembled questionnaire was evaluated through the squared correlation between estimated and true parameters, the root mean square error between the estimated and true parameters, the average difference between the estimated and true inter-trait correlations, and the average standard error for each trait level. The proposed GA offered more accurate trait estimates than the BF search within a reasonable computation time in every simulation condition. Such improvements were especially important when measuring correlated traits and when the relative item pool sizes were higher. A user-friendly online implementation of the algorithm was made available to the users. Springer US 2021-09-09 2022 /pmc/articles/PMC9170671/ /pubmed/34505277 http://dx.doi.org/10.3758/s13428-021-01677-4 Text en © The Author(s) 2021 https://creativecommons.org/licenses/by/4.0/Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ (https://creativecommons.org/licenses/by/4.0/) . |
spellingShingle | Article Kreitchmann, Rodrigo Schames Abad, Francisco J. Sorrel, Miguel A. A genetic algorithm for optimal assembly of pairwise forced-choice questionnaires |
title | A genetic algorithm for optimal assembly of pairwise forced-choice questionnaires |
title_full | A genetic algorithm for optimal assembly of pairwise forced-choice questionnaires |
title_fullStr | A genetic algorithm for optimal assembly of pairwise forced-choice questionnaires |
title_full_unstemmed | A genetic algorithm for optimal assembly of pairwise forced-choice questionnaires |
title_short | A genetic algorithm for optimal assembly of pairwise forced-choice questionnaires |
title_sort | genetic algorithm for optimal assembly of pairwise forced-choice questionnaires |
topic | Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9170671/ https://www.ncbi.nlm.nih.gov/pubmed/34505277 http://dx.doi.org/10.3758/s13428-021-01677-4 |
work_keys_str_mv | AT kreitchmannrodrigoschames ageneticalgorithmforoptimalassemblyofpairwiseforcedchoicequestionnaires AT abadfranciscoj ageneticalgorithmforoptimalassemblyofpairwiseforcedchoicequestionnaires AT sorrelmiguela ageneticalgorithmforoptimalassemblyofpairwiseforcedchoicequestionnaires AT kreitchmannrodrigoschames geneticalgorithmforoptimalassemblyofpairwiseforcedchoicequestionnaires AT abadfranciscoj geneticalgorithmforoptimalassemblyofpairwiseforcedchoicequestionnaires AT sorrelmiguela geneticalgorithmforoptimalassemblyofpairwiseforcedchoicequestionnaires |