Cargando…
Comparing the Similarity of Responses Received from Studies in Amazon’s Mechanical Turk to Studies Conducted Online and with Direct Recruitment
Computer and internet based questionnaires have become a standard tool in Human-Computer Interaction research and other related fields, such as psychology and sociology. Amazon’s Mechanical Turk (AMT) service is a new method of recruiting participants and conducting certain types of experiments. Thi...
Autores principales: | , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Public Library of Science
2015
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4397064/ https://www.ncbi.nlm.nih.gov/pubmed/25876027 http://dx.doi.org/10.1371/journal.pone.0121595 |
_version_ | 1782366667985125376 |
---|---|
author | Bartneck, Christoph Duenser, Andreas Moltchanova, Elena Zawieska, Karolina |
author_facet | Bartneck, Christoph Duenser, Andreas Moltchanova, Elena Zawieska, Karolina |
author_sort | Bartneck, Christoph |
collection | PubMed |
description | Computer and internet based questionnaires have become a standard tool in Human-Computer Interaction research and other related fields, such as psychology and sociology. Amazon’s Mechanical Turk (AMT) service is a new method of recruiting participants and conducting certain types of experiments. This study compares whether participants recruited through AMT give different responses than participants recruited through an online forum or recruited directly on a university campus. Moreover, we compare whether a study conducted within AMT results in different responses compared to a study for which participants are recruited through AMT but which is conducted using an external online questionnaire service. The results of this study show that there is a statistical difference between results obtained from participants recruited through AMT compared to the results from the participant recruited on campus or through online forums. We do, however, argue that this difference is so small that it has no practical consequence. There was no significant difference between running the study within AMT compared to running it with an online questionnaire service. There was no significant difference between results obtained directly from within AMT compared to results obtained in the campus and online forum condition. This may suggest that AMT is a viable and economical option for recruiting participants and for conducting studies as setting up and running a study with AMT generally requires less effort and time compared to other frequently used methods. We discuss our findings as well as limitations of using AMT for empirical studies. |
format | Online Article Text |
id | pubmed-4397064 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2015 |
publisher | Public Library of Science |
record_format | MEDLINE/PubMed |
spelling | pubmed-43970642015-04-21 Comparing the Similarity of Responses Received from Studies in Amazon’s Mechanical Turk to Studies Conducted Online and with Direct Recruitment Bartneck, Christoph Duenser, Andreas Moltchanova, Elena Zawieska, Karolina PLoS One Research Article Computer and internet based questionnaires have become a standard tool in Human-Computer Interaction research and other related fields, such as psychology and sociology. Amazon’s Mechanical Turk (AMT) service is a new method of recruiting participants and conducting certain types of experiments. This study compares whether participants recruited through AMT give different responses than participants recruited through an online forum or recruited directly on a university campus. Moreover, we compare whether a study conducted within AMT results in different responses compared to a study for which participants are recruited through AMT but which is conducted using an external online questionnaire service. The results of this study show that there is a statistical difference between results obtained from participants recruited through AMT compared to the results from the participant recruited on campus or through online forums. We do, however, argue that this difference is so small that it has no practical consequence. There was no significant difference between running the study within AMT compared to running it with an online questionnaire service. There was no significant difference between results obtained directly from within AMT compared to results obtained in the campus and online forum condition. This may suggest that AMT is a viable and economical option for recruiting participants and for conducting studies as setting up and running a study with AMT generally requires less effort and time compared to other frequently used methods. We discuss our findings as well as limitations of using AMT for empirical studies. Public Library of Science 2015-04-14 /pmc/articles/PMC4397064/ /pubmed/25876027 http://dx.doi.org/10.1371/journal.pone.0121595 Text en © 2015 Bartneck et al http://creativecommons.org/licenses/by/4.0/ This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are properly credited. |
spellingShingle | Research Article Bartneck, Christoph Duenser, Andreas Moltchanova, Elena Zawieska, Karolina Comparing the Similarity of Responses Received from Studies in Amazon’s Mechanical Turk to Studies Conducted Online and with Direct Recruitment |
title | Comparing the Similarity of Responses Received from Studies in Amazon’s Mechanical Turk to Studies Conducted Online and with Direct Recruitment |
title_full | Comparing the Similarity of Responses Received from Studies in Amazon’s Mechanical Turk to Studies Conducted Online and with Direct Recruitment |
title_fullStr | Comparing the Similarity of Responses Received from Studies in Amazon’s Mechanical Turk to Studies Conducted Online and with Direct Recruitment |
title_full_unstemmed | Comparing the Similarity of Responses Received from Studies in Amazon’s Mechanical Turk to Studies Conducted Online and with Direct Recruitment |
title_short | Comparing the Similarity of Responses Received from Studies in Amazon’s Mechanical Turk to Studies Conducted Online and with Direct Recruitment |
title_sort | comparing the similarity of responses received from studies in amazon’s mechanical turk to studies conducted online and with direct recruitment |
topic | Research Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4397064/ https://www.ncbi.nlm.nih.gov/pubmed/25876027 http://dx.doi.org/10.1371/journal.pone.0121595 |
work_keys_str_mv | AT bartneckchristoph comparingthesimilarityofresponsesreceivedfromstudiesinamazonsmechanicalturktostudiesconductedonlineandwithdirectrecruitment AT duenserandreas comparingthesimilarityofresponsesreceivedfromstudiesinamazonsmechanicalturktostudiesconductedonlineandwithdirectrecruitment AT moltchanovaelena comparingthesimilarityofresponsesreceivedfromstudiesinamazonsmechanicalturktostudiesconductedonlineandwithdirectrecruitment AT zawieskakarolina comparingthesimilarityofresponsesreceivedfromstudiesinamazonsmechanicalturktostudiesconductedonlineandwithdirectrecruitment |