Crowdsourcing Subjective Annotations Using Pairwise Comparisons Reduces Bias and Error Compared to the Majority-vote Method

Loading...
Thumbnail Image
Access rights
openAccess
Journal Title
Journal ISSN
Volume Title
A1 Alkuperäisartikkeli tieteellisessä aikakauslehdessä
Date
2023-10-04
Major/Subject
Mcode
Degree programme
Language
en
Pages
Series
Proceedings of the ACM on Human-Computer Interaction, Volume 7, issue CSCW2
Abstract
How to better reduce measurement variability and bias introduced by subjectivity in crowdsourced labelling remains an open question. We introduce a theoretical framework for understanding how random error and measurement bias enter into crowdsourced annotations of subjective constructs. We then propose a pipeline that combines pairwise comparison labelling with Elo scoring, and demonstrate that it outperforms the ubiquitous majority-voting method in reducing both types of measurement error. To assess the performance of the labelling approaches, we constructed an agent-based model of crowdsourced labelling that lets us introduce different types of subjectivity into the tasks. We find that under most conditions with task subjectivity, the comparison approach produced higher f1 scores. Further, the comparison approach is less susceptible to inflating bias, which majority voting tends to do. To facilitate applications, we show with simulated and real-world data that the number of required random comparisons for the same classification accuracy scales log-linearly O(N log N) with the number of labelled items. We also implemented the Elo system as an open-source Python package.
Description
Funding Information: This project is supported by Academy of Finland grants 320780 and 320781 Publisher Copyright: © 2023 Owner/Author.
Keywords
comparison method, crowdsourcing, majority-vote method, subjectivity
Other note
Citation
Narimanzadeh , H , Badie-Modiri , A , Smirnova , I G & Chen , T H Y 2023 , ' Crowdsourcing Subjective Annotations Using Pairwise Comparisons Reduces Bias and Error Compared to the Majority-vote Method ' , Proceedings of the ACM on Human-Computer Interaction , vol. 7 , no. CSCW2 , 3610183 . https://doi.org/10.1145/3610183