Cargando…
Data Quality and Study Compliance Among College Students Across 2 Recruitment Sources: Two Study Investigation
BACKGROUND: Models of satisficing suggest that study participants may not fully process survey items and provide accurate responses when survey burden is higher and when participant motivation is lower. Participants who do not fully process survey instructions can reduce a study’s power and hinder g...
Autores principales: | , , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
JMIR Publications
2022
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9789498/ https://www.ncbi.nlm.nih.gov/pubmed/36485020 http://dx.doi.org/10.2196/39488 |
_version_ | 1784858968724602880 |
---|---|
author | Braitman, Abby L Strowger, Megan Shipley, Jennifer L Ortman, Jordan MacIntyre, Rachel I Bauer, Elizabeth A |
author_facet | Braitman, Abby L Strowger, Megan Shipley, Jennifer L Ortman, Jordan MacIntyre, Rachel I Bauer, Elizabeth A |
author_sort | Braitman, Abby L |
collection | PubMed |
description | BACKGROUND: Models of satisficing suggest that study participants may not fully process survey items and provide accurate responses when survey burden is higher and when participant motivation is lower. Participants who do not fully process survey instructions can reduce a study’s power and hinder generalizability. Common concerns among researchers using self-report measures are data quality and participant compliance. Similarly, attrition can hurt the power and generalizability of a study. OBJECTIVE: Given that college students comprise most samples in psychological studies, especially examinations of student issues and psychological health, it is critical to understand how college student recruitment sources impact data quality (operationalized as attention check items with directive instructions and correct answers) and retention (operationalized as the completion of follow-up surveys over time). This examination aimed to examine the following: whether data quality varies across recruitment sources, whether study retention varies across recruitment sources, the impact of data quality on study variable associations, the impact of data quality on measures of internal consistency, and whether the demographic qualities of participants significantly vary across those who failed attention checks versus those who did not. METHODS: This examination was a follow-up analysis of 2 previously published studies to explore data quality and study compliance. Study 1 was a cross-sectional, web-based survey examining college stressors and psychological health (282/407, 69.3% female; 230/407, 56.5% White, 113/407, 27.8% Black; mean age 22.65, SD 6.73 years). Study 2 was a longitudinal college drinking intervention trial with an in-person baseline session and 2 web-based follow-up surveys (378/528, 71.6% female; 213/528, 40.3% White, 277/528, 52.5% Black; mean age 19.85, SD 1.65 years). Attention checks were included in both studies to assess data quality. Participants for both studies were recruited from a psychology participation pool (a pull-in method; for course credit) and the general student body (a push-out method; for monetary payment or raffle entry). RESULTS: A greater proportion of participants recruited through the psychology pool failed attention checks in both studies, suggesting poorer data quality. The psychology pool was also associated with lower retention rates over time. After screening out those who failed attention checks, some correlations among the study variables were stronger, some were weaker, and some were fairly similar, potentially suggesting bias introduced by including these participants. Differences among the indicators of internal consistency for the study measures were negligible. Finally, attention check failure was not significantly associated with most demographic characteristics but varied across some racial identities. This suggests that filtering out data from participants who failed attention checks may not limit sample diversity. CONCLUSIONS: Investigators conducting college student research should carefully consider recruitment and include attention checks or other means of detecting poor quality data. Recommendations for researchers are discussed. |
format | Online Article Text |
id | pubmed-9789498 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2022 |
publisher | JMIR Publications |
record_format | MEDLINE/PubMed |
spelling | pubmed-97894982022-12-25 Data Quality and Study Compliance Among College Students Across 2 Recruitment Sources: Two Study Investigation Braitman, Abby L Strowger, Megan Shipley, Jennifer L Ortman, Jordan MacIntyre, Rachel I Bauer, Elizabeth A JMIR Form Res Original Paper BACKGROUND: Models of satisficing suggest that study participants may not fully process survey items and provide accurate responses when survey burden is higher and when participant motivation is lower. Participants who do not fully process survey instructions can reduce a study’s power and hinder generalizability. Common concerns among researchers using self-report measures are data quality and participant compliance. Similarly, attrition can hurt the power and generalizability of a study. OBJECTIVE: Given that college students comprise most samples in psychological studies, especially examinations of student issues and psychological health, it is critical to understand how college student recruitment sources impact data quality (operationalized as attention check items with directive instructions and correct answers) and retention (operationalized as the completion of follow-up surveys over time). This examination aimed to examine the following: whether data quality varies across recruitment sources, whether study retention varies across recruitment sources, the impact of data quality on study variable associations, the impact of data quality on measures of internal consistency, and whether the demographic qualities of participants significantly vary across those who failed attention checks versus those who did not. METHODS: This examination was a follow-up analysis of 2 previously published studies to explore data quality and study compliance. Study 1 was a cross-sectional, web-based survey examining college stressors and psychological health (282/407, 69.3% female; 230/407, 56.5% White, 113/407, 27.8% Black; mean age 22.65, SD 6.73 years). Study 2 was a longitudinal college drinking intervention trial with an in-person baseline session and 2 web-based follow-up surveys (378/528, 71.6% female; 213/528, 40.3% White, 277/528, 52.5% Black; mean age 19.85, SD 1.65 years). Attention checks were included in both studies to assess data quality. Participants for both studies were recruited from a psychology participation pool (a pull-in method; for course credit) and the general student body (a push-out method; for monetary payment or raffle entry). RESULTS: A greater proportion of participants recruited through the psychology pool failed attention checks in both studies, suggesting poorer data quality. The psychology pool was also associated with lower retention rates over time. After screening out those who failed attention checks, some correlations among the study variables were stronger, some were weaker, and some were fairly similar, potentially suggesting bias introduced by including these participants. Differences among the indicators of internal consistency for the study measures were negligible. Finally, attention check failure was not significantly associated with most demographic characteristics but varied across some racial identities. This suggests that filtering out data from participants who failed attention checks may not limit sample diversity. CONCLUSIONS: Investigators conducting college student research should carefully consider recruitment and include attention checks or other means of detecting poor quality data. Recommendations for researchers are discussed. JMIR Publications 2022-12-09 /pmc/articles/PMC9789498/ /pubmed/36485020 http://dx.doi.org/10.2196/39488 Text en ©Abby L Braitman, Megan Strowger, Jennifer L Shipley, Jordan Ortman, Rachel I MacIntyre, Elizabeth A Bauer. Originally published in JMIR Formative Research (https://formative.jmir.org), 09.12.2022. https://creativecommons.org/licenses/by/4.0/This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included. |
spellingShingle | Original Paper Braitman, Abby L Strowger, Megan Shipley, Jennifer L Ortman, Jordan MacIntyre, Rachel I Bauer, Elizabeth A Data Quality and Study Compliance Among College Students Across 2 Recruitment Sources: Two Study Investigation |
title | Data Quality and Study Compliance Among College Students Across 2 Recruitment Sources: Two Study Investigation |
title_full | Data Quality and Study Compliance Among College Students Across 2 Recruitment Sources: Two Study Investigation |
title_fullStr | Data Quality and Study Compliance Among College Students Across 2 Recruitment Sources: Two Study Investigation |
title_full_unstemmed | Data Quality and Study Compliance Among College Students Across 2 Recruitment Sources: Two Study Investigation |
title_short | Data Quality and Study Compliance Among College Students Across 2 Recruitment Sources: Two Study Investigation |
title_sort | data quality and study compliance among college students across 2 recruitment sources: two study investigation |
topic | Original Paper |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9789498/ https://www.ncbi.nlm.nih.gov/pubmed/36485020 http://dx.doi.org/10.2196/39488 |
work_keys_str_mv | AT braitmanabbyl dataqualityandstudycomplianceamongcollegestudentsacross2recruitmentsourcestwostudyinvestigation AT strowgermegan dataqualityandstudycomplianceamongcollegestudentsacross2recruitmentsourcestwostudyinvestigation AT shipleyjenniferl dataqualityandstudycomplianceamongcollegestudentsacross2recruitmentsourcestwostudyinvestigation AT ortmanjordan dataqualityandstudycomplianceamongcollegestudentsacross2recruitmentsourcestwostudyinvestigation AT macintyreracheli dataqualityandstudycomplianceamongcollegestudentsacross2recruitmentsourcestwostudyinvestigation AT bauerelizabetha dataqualityandstudycomplianceamongcollegestudentsacross2recruitmentsourcestwostudyinvestigation |