Cargando…

Data Quality and Study Compliance Among College Students Across 2 Recruitment Sources: Two Study Investigation

BACKGROUND: Models of satisficing suggest that study participants may not fully process survey items and provide accurate responses when survey burden is higher and when participant motivation is lower. Participants who do not fully process survey instructions can reduce a study’s power and hinder g...

Descripción completa

Detalles Bibliográficos
Autores principales: Braitman, Abby L, Strowger, Megan, Shipley, Jennifer L, Ortman, Jordan, MacIntyre, Rachel I, Bauer, Elizabeth A
Formato: Online Artículo Texto
Lenguaje:English
Publicado: JMIR Publications 2022
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9789498/
https://www.ncbi.nlm.nih.gov/pubmed/36485020
http://dx.doi.org/10.2196/39488
_version_ 1784858968724602880
author Braitman, Abby L
Strowger, Megan
Shipley, Jennifer L
Ortman, Jordan
MacIntyre, Rachel I
Bauer, Elizabeth A
author_facet Braitman, Abby L
Strowger, Megan
Shipley, Jennifer L
Ortman, Jordan
MacIntyre, Rachel I
Bauer, Elizabeth A
author_sort Braitman, Abby L
collection PubMed
description BACKGROUND: Models of satisficing suggest that study participants may not fully process survey items and provide accurate responses when survey burden is higher and when participant motivation is lower. Participants who do not fully process survey instructions can reduce a study’s power and hinder generalizability. Common concerns among researchers using self-report measures are data quality and participant compliance. Similarly, attrition can hurt the power and generalizability of a study. OBJECTIVE: Given that college students comprise most samples in psychological studies, especially examinations of student issues and psychological health, it is critical to understand how college student recruitment sources impact data quality (operationalized as attention check items with directive instructions and correct answers) and retention (operationalized as the completion of follow-up surveys over time). This examination aimed to examine the following: whether data quality varies across recruitment sources, whether study retention varies across recruitment sources, the impact of data quality on study variable associations, the impact of data quality on measures of internal consistency, and whether the demographic qualities of participants significantly vary across those who failed attention checks versus those who did not. METHODS: This examination was a follow-up analysis of 2 previously published studies to explore data quality and study compliance. Study 1 was a cross-sectional, web-based survey examining college stressors and psychological health (282/407, 69.3% female; 230/407, 56.5% White, 113/407, 27.8% Black; mean age 22.65, SD 6.73 years). Study 2 was a longitudinal college drinking intervention trial with an in-person baseline session and 2 web-based follow-up surveys (378/528, 71.6% female; 213/528, 40.3% White, 277/528, 52.5% Black; mean age 19.85, SD 1.65 years). Attention checks were included in both studies to assess data quality. Participants for both studies were recruited from a psychology participation pool (a pull-in method; for course credit) and the general student body (a push-out method; for monetary payment or raffle entry). RESULTS: A greater proportion of participants recruited through the psychology pool failed attention checks in both studies, suggesting poorer data quality. The psychology pool was also associated with lower retention rates over time. After screening out those who failed attention checks, some correlations among the study variables were stronger, some were weaker, and some were fairly similar, potentially suggesting bias introduced by including these participants. Differences among the indicators of internal consistency for the study measures were negligible. Finally, attention check failure was not significantly associated with most demographic characteristics but varied across some racial identities. This suggests that filtering out data from participants who failed attention checks may not limit sample diversity. CONCLUSIONS: Investigators conducting college student research should carefully consider recruitment and include attention checks or other means of detecting poor quality data. Recommendations for researchers are discussed.
format Online
Article
Text
id pubmed-9789498
institution National Center for Biotechnology Information
language English
publishDate 2022
publisher JMIR Publications
record_format MEDLINE/PubMed
spelling pubmed-97894982022-12-25 Data Quality and Study Compliance Among College Students Across 2 Recruitment Sources: Two Study Investigation Braitman, Abby L Strowger, Megan Shipley, Jennifer L Ortman, Jordan MacIntyre, Rachel I Bauer, Elizabeth A JMIR Form Res Original Paper BACKGROUND: Models of satisficing suggest that study participants may not fully process survey items and provide accurate responses when survey burden is higher and when participant motivation is lower. Participants who do not fully process survey instructions can reduce a study’s power and hinder generalizability. Common concerns among researchers using self-report measures are data quality and participant compliance. Similarly, attrition can hurt the power and generalizability of a study. OBJECTIVE: Given that college students comprise most samples in psychological studies, especially examinations of student issues and psychological health, it is critical to understand how college student recruitment sources impact data quality (operationalized as attention check items with directive instructions and correct answers) and retention (operationalized as the completion of follow-up surveys over time). This examination aimed to examine the following: whether data quality varies across recruitment sources, whether study retention varies across recruitment sources, the impact of data quality on study variable associations, the impact of data quality on measures of internal consistency, and whether the demographic qualities of participants significantly vary across those who failed attention checks versus those who did not. METHODS: This examination was a follow-up analysis of 2 previously published studies to explore data quality and study compliance. Study 1 was a cross-sectional, web-based survey examining college stressors and psychological health (282/407, 69.3% female; 230/407, 56.5% White, 113/407, 27.8% Black; mean age 22.65, SD 6.73 years). Study 2 was a longitudinal college drinking intervention trial with an in-person baseline session and 2 web-based follow-up surveys (378/528, 71.6% female; 213/528, 40.3% White, 277/528, 52.5% Black; mean age 19.85, SD 1.65 years). Attention checks were included in both studies to assess data quality. Participants for both studies were recruited from a psychology participation pool (a pull-in method; for course credit) and the general student body (a push-out method; for monetary payment or raffle entry). RESULTS: A greater proportion of participants recruited through the psychology pool failed attention checks in both studies, suggesting poorer data quality. The psychology pool was also associated with lower retention rates over time. After screening out those who failed attention checks, some correlations among the study variables were stronger, some were weaker, and some were fairly similar, potentially suggesting bias introduced by including these participants. Differences among the indicators of internal consistency for the study measures were negligible. Finally, attention check failure was not significantly associated with most demographic characteristics but varied across some racial identities. This suggests that filtering out data from participants who failed attention checks may not limit sample diversity. CONCLUSIONS: Investigators conducting college student research should carefully consider recruitment and include attention checks or other means of detecting poor quality data. Recommendations for researchers are discussed. JMIR Publications 2022-12-09 /pmc/articles/PMC9789498/ /pubmed/36485020 http://dx.doi.org/10.2196/39488 Text en ©Abby L Braitman, Megan Strowger, Jennifer L Shipley, Jordan Ortman, Rachel I MacIntyre, Elizabeth A Bauer. Originally published in JMIR Formative Research (https://formative.jmir.org), 09.12.2022. https://creativecommons.org/licenses/by/4.0/This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.
spellingShingle Original Paper
Braitman, Abby L
Strowger, Megan
Shipley, Jennifer L
Ortman, Jordan
MacIntyre, Rachel I
Bauer, Elizabeth A
Data Quality and Study Compliance Among College Students Across 2 Recruitment Sources: Two Study Investigation
title Data Quality and Study Compliance Among College Students Across 2 Recruitment Sources: Two Study Investigation
title_full Data Quality and Study Compliance Among College Students Across 2 Recruitment Sources: Two Study Investigation
title_fullStr Data Quality and Study Compliance Among College Students Across 2 Recruitment Sources: Two Study Investigation
title_full_unstemmed Data Quality and Study Compliance Among College Students Across 2 Recruitment Sources: Two Study Investigation
title_short Data Quality and Study Compliance Among College Students Across 2 Recruitment Sources: Two Study Investigation
title_sort data quality and study compliance among college students across 2 recruitment sources: two study investigation
topic Original Paper
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9789498/
https://www.ncbi.nlm.nih.gov/pubmed/36485020
http://dx.doi.org/10.2196/39488
work_keys_str_mv AT braitmanabbyl dataqualityandstudycomplianceamongcollegestudentsacross2recruitmentsourcestwostudyinvestigation
AT strowgermegan dataqualityandstudycomplianceamongcollegestudentsacross2recruitmentsourcestwostudyinvestigation
AT shipleyjenniferl dataqualityandstudycomplianceamongcollegestudentsacross2recruitmentsourcestwostudyinvestigation
AT ortmanjordan dataqualityandstudycomplianceamongcollegestudentsacross2recruitmentsourcestwostudyinvestigation
AT macintyreracheli dataqualityandstudycomplianceamongcollegestudentsacross2recruitmentsourcestwostudyinvestigation
AT bauerelizabetha dataqualityandstudycomplianceamongcollegestudentsacross2recruitmentsourcestwostudyinvestigation