Statistical Significance Testing at CHI PLAY: Challenges and Opportunities for More Transparency

Thumbnail Image
Journal Title
Journal ISSN
Volume Title
A4 Artikkeli konferenssijulkaisussa
This publication is imported from Aalto University research portal.
View publication in the Research portal
View/Open full text file from the Research portal
Other link related to publication
Author
Date
2020-11-02
Department
Department of Computer Science
Prof. Mekler Elisa
Major/Subject
Mcode
Degree programme
Language
en
Pages
4-18
Series
CHI PLAY '20: Proceedings of the Annual Symposium on Computer-Human Interaction in Play
Abstract
Statistical Significance Testing -- or Null Hypothesis Significance Testing (NHST) -- is common to quantitative CHI PLAY research. Drawing from recent work in HCI and psychology promoting transparent statistics and the reduction of questionable research practices, we systematically review the reporting quality of 119 CHI PLAY papers using NHST (data and analysis plan at https://osf.io/4mcbn/. We find that over half of these papers employ NHST without specific statistical hypotheses or research questions, which may risk the proliferation of false positive findings. Moreover, we observe inconsistencies in the reporting of sample sizes and statistical tests. These issues reflect fundamental incompatibilities between NHST and the frequently exploratory work common to CHI PLAY. We discuss the complementary roles of exploratory and confirmatory research, and provide a template for more transparent research and reporting practices.
Description
Keywords
DOI
10.1145/3410404.3414229
Citation
Vornhagen , J , Tyack , A & Mekler , E 2020 , Statistical Significance Testing at CHI PLAY: Challenges and Opportunities for More Transparency . in CHI PLAY 2020 - Proceedings of the Annual Symposium on Computer-Human Interaction in Play . ACM , pp. 4-18 , ACM SIGCHI Annual Symposium on Computer-Human Interaction in Play , Virtual, Online , Canada , 01/11/2020 . https://doi.org/10.1145/3410404.3414229
Permanent link to this item
https://urn.fi/URN:NBN:fi:aalto-2020113020606
Collections
[article-cris] Perustieteiden korkeakoulu / SCI