A genetic algorithm for optimal assembly of pairwise forced-choice questionnaires
The use of multidimensional forced-choice questionnaires has been proposed as a means of improving validity in the assessment of non-cognitive attributes in high-stakes scenarios. However, the reduced precision of trait estimates in this questionnaire format is an important drawback. Accordingly, th...
| Autores: | , , |
|---|---|
| Tipo de recurso: | artículo |
| Fecha de publicación: | 2021 |
| País: | España |
| Institución: | Universidad Autónoma de Madrid |
| Repositorio: | Biblos-e Archivo. Repositorio Institucional de la UAM |
| Idioma: | inglés |
| OAI Identifier: | oai:repositorio.uam.es:10486/700723 |
| Acceso en línea: | http://hdl.handle.net/10486/700723 https://dx.doi.org/10.3758/s13428-021-01677-4 |
| Access Level: | acceso abierto |
| Palabra clave: | forced-choice format genetic algorithms ipsative data multidimensional item response theory reliability test assembly Psicología |
| id |
ES_79d35bee8f9dfe9f85cb230dc281f776 |
|---|---|
| oai_identifier_str |
oai:repositorio.uam.es:10486/700723 |
| network_acronym_str |
ES |
| network_name_str |
España |
| repository_id_str |
|
| spelling |
A genetic algorithm for optimal assembly of pairwise forced-choice questionnairesKreitchmann, Rodrigo SchamesAbad Santos, FranciscoSorrel Luján, Miguel Ángelforced-choice formatgenetic algorithmsipsative datamultidimensional item response theoryreliabilitytest assemblyPsicologíaThe use of multidimensional forced-choice questionnaires has been proposed as a means of improving validity in the assessment of non-cognitive attributes in high-stakes scenarios. However, the reduced precision of trait estimates in this questionnaire format is an important drawback. Accordingly, this article presents an optimization procedure for assembling pairwise forced-choice questionnaires while maximizing posterior marginal reliabilities. This procedure is performed through the adaptation of a known genetic algorithm (GA) for combinatorial problems. In a simulation study, the efficiency of the proposed procedure was compared with a quasi-brute-force (BF) search. For this purpose, five-dimensional item pools were simulated to emulate the real problem of generating a forced-choice personality questionnaire under the five-factor model. Three factors were manipulated: (1) the length of the questionnaire, (2) the relative item pool size with respect to the questionnaire’s length, and (3) the true correlations between traits. The recovery of the person parameters for each assembled questionnaire was evaluated through the squared correlation between estimated and true parameters, the root mean square error between the estimated and true parameters, the average difference between the estimated and true inter-trait correlations, and the average standard error for each trait level. The proposed GA offered more accurate trait estimates than the BF search within a reasonable computation time in every simulation condition. Such improvements were especially important when measuring correlated traits and when the relative item pool sizes were higher. A user-friendly online implementation of the algorithm was made available to the usersSpringerDepartamento de Psicología Social y MetodologíaFacultad de Psicología20212021-09-09research articlehttp://purl.org/coar/resource_type/c_2df8fbb1VoRhttp://purl.org/coar/version/c_970fb48d4fbd8a85info:eu-repo/semantics/articleapplication/pdfhttp://hdl.handle.net/10486/700723https://dx.doi.org/10.3758/s13428-021-01677-4reponame:Biblos-e Archivo. Repositorio Institucional de la UAMinstname:Universidad Autónoma de MadridInglésengopen accesshttp://purl.org/coar/access_right/c_abf2info:eu-repo/semantics/openAccessoai:repositorio.uam.es:10486/7007232026-06-23T12:46:27Z |
| dc.title.none.fl_str_mv |
A genetic algorithm for optimal assembly of pairwise forced-choice questionnaires |
| title |
A genetic algorithm for optimal assembly of pairwise forced-choice questionnaires |
| spellingShingle |
A genetic algorithm for optimal assembly of pairwise forced-choice questionnaires Kreitchmann, Rodrigo Schames forced-choice format genetic algorithms ipsative data multidimensional item response theory reliability test assembly Psicología |
| title_short |
A genetic algorithm for optimal assembly of pairwise forced-choice questionnaires |
| title_full |
A genetic algorithm for optimal assembly of pairwise forced-choice questionnaires |
| title_fullStr |
A genetic algorithm for optimal assembly of pairwise forced-choice questionnaires |
| title_full_unstemmed |
A genetic algorithm for optimal assembly of pairwise forced-choice questionnaires |
| title_sort |
A genetic algorithm for optimal assembly of pairwise forced-choice questionnaires |
| dc.creator.none.fl_str_mv |
Kreitchmann, Rodrigo Schames Abad Santos, Francisco Sorrel Luján, Miguel Ángel |
| author |
Kreitchmann, Rodrigo Schames |
| author_facet |
Kreitchmann, Rodrigo Schames Abad Santos, Francisco Sorrel Luján, Miguel Ángel |
| author_role |
author |
| author2 |
Abad Santos, Francisco Sorrel Luján, Miguel Ángel |
| author2_role |
author author |
| dc.contributor.none.fl_str_mv |
Departamento de Psicología Social y Metodología Facultad de Psicología |
| dc.subject.none.fl_str_mv |
forced-choice format genetic algorithms ipsative data multidimensional item response theory reliability test assembly Psicología |
| topic |
forced-choice format genetic algorithms ipsative data multidimensional item response theory reliability test assembly Psicología |
| description |
The use of multidimensional forced-choice questionnaires has been proposed as a means of improving validity in the assessment of non-cognitive attributes in high-stakes scenarios. However, the reduced precision of trait estimates in this questionnaire format is an important drawback. Accordingly, this article presents an optimization procedure for assembling pairwise forced-choice questionnaires while maximizing posterior marginal reliabilities. This procedure is performed through the adaptation of a known genetic algorithm (GA) for combinatorial problems. In a simulation study, the efficiency of the proposed procedure was compared with a quasi-brute-force (BF) search. For this purpose, five-dimensional item pools were simulated to emulate the real problem of generating a forced-choice personality questionnaire under the five-factor model. Three factors were manipulated: (1) the length of the questionnaire, (2) the relative item pool size with respect to the questionnaire’s length, and (3) the true correlations between traits. The recovery of the person parameters for each assembled questionnaire was evaluated through the squared correlation between estimated and true parameters, the root mean square error between the estimated and true parameters, the average difference between the estimated and true inter-trait correlations, and the average standard error for each trait level. The proposed GA offered more accurate trait estimates than the BF search within a reasonable computation time in every simulation condition. Such improvements were especially important when measuring correlated traits and when the relative item pool sizes were higher. A user-friendly online implementation of the algorithm was made available to the users |
| publishDate |
2021 |
| dc.date.none.fl_str_mv |
2021 2021-09-09 |
| dc.type.none.fl_str_mv |
research article http://purl.org/coar/resource_type/c_2df8fbb1 VoR http://purl.org/coar/version/c_970fb48d4fbd8a85 |
| dc.type.openaire.fl_str_mv |
info:eu-repo/semantics/article |
| format |
article |
| dc.identifier.none.fl_str_mv |
http://hdl.handle.net/10486/700723 https://dx.doi.org/10.3758/s13428-021-01677-4 |
| url |
http://hdl.handle.net/10486/700723 https://dx.doi.org/10.3758/s13428-021-01677-4 |
| dc.language.none.fl_str_mv |
Inglés eng |
| language_invalid_str_mv |
Inglés |
| language |
eng |
| dc.rights.none.fl_str_mv |
open access http://purl.org/coar/access_right/c_abf2 |
| dc.rights.openaire.fl_str_mv |
info:eu-repo/semantics/openAccess |
| rights_invalid_str_mv |
open access http://purl.org/coar/access_right/c_abf2 |
| eu_rights_str_mv |
openAccess |
| dc.format.none.fl_str_mv |
application/pdf |
| dc.publisher.none.fl_str_mv |
Springer |
| publisher.none.fl_str_mv |
Springer |
| dc.source.none.fl_str_mv |
reponame:Biblos-e Archivo. Repositorio Institucional de la UAM instname:Universidad Autónoma de Madrid |
| instname_str |
Universidad Autónoma de Madrid |
| reponame_str |
Biblos-e Archivo. Repositorio Institucional de la UAM |
| collection |
Biblos-e Archivo. Repositorio Institucional de la UAM |
| repository.name.fl_str_mv |
|
| repository.mail.fl_str_mv |
|
| _version_ |
1869411391575687168 |
| score |
15.223283 |