Crowdsourcing in Computing Education Research: Case Amazon MTurk
Loading...
Access rights
openAccess
acceptedVersion
URL
Journal Title
Journal ISSN
Volume Title
A4 Artikkeli konferenssijulkaisussa
This publication is imported from Aalto University research portal.
View publication in the Research portal (opens in new window)
View/Open full text file from the Research portal (opens in new window)
Other link related to publication (opens in new window)
View publication in the Research portal (opens in new window)
View/Open full text file from the Research portal (opens in new window)
Other link related to publication (opens in new window)
Date
2020-11-19
Department
Major/Subject
Mcode
Degree programme
Language
en
Pages
Series
Koli Calling '20: Proceedings of the 20th Koli Calling International Conference on Computing Education Research, pp. 1-5
Abstract
Crowdsourcing platforms such as Amazon MTurk provide access to a human workforce that can be given tasks to complete online for a fee. In this article, we review studies in computing education research (CER) that rely on crowdsourcing; we also describe our own experiences of using Amazon MTurk for a CER study. We discuss challenges in recruiting workers with specific backgrounds—such as no programming experience—and considerations in filtering out unreliable research participants. Combining recommendations from the literature with the lessons that we learned whilst conducting our study, we synthesize advice for researchers in CER who are considering crowdsourcing. In our case study, we did not find widespread foul play by crowdsourced workers and, overall, our experiences and the literature suggest that crowdsourced CER is feasible. It is, however, uncertain to what extent crowdsourced data can produce answers that apply to specific educational contexts. More research is needed before definitive conclusions can be drawn about the validity and generalizability of crowdsourced CER.Description
Keywords
Crowdsourcing, CER
Other note
Citation
Hellas, A, Zavgorodniaia, A & Sorva, J 2020, Crowdsourcing in Computing Education Research: Case Amazon MTurk . in Koli Calling '20: Proceedings of the 20th Koli Calling International Conference on Computing Education Research ., 32, ACM, pp. 1-5, Koli Calling - International Conference on Computing Education Research, Virtual, Online, Finland, 19/11/2020 . https://doi.org/10.1145/3428029.3428062