User Modelling for Avoiding Overfitting in Interactive Knowledge Elicitation for Prediction

Thumbnail Image

Access rights

openAccess
preprint

URL

Journal Title

Journal ISSN

Volume Title

A4 Artikkeli konferenssijulkaisussa
This publication is imported from Aalto University research portal.
View publication in the Research portal (opens in new window)
View/Open full text file from the Research portal (opens in new window)
Other link related to publication (opens in new window)

Authors

Date

2018-03-08

Department

School services,SCI
Department of Computer Science

Major/Subject

Mcode

Degree programme

Language

en

Pages

6

Series

IUI 2018 - Proceedings of the 23rd International Conference on Intelligent User Interfaces, pp. 305-310

Abstract

In human-in-the-loop machine learning, the user provides information beyond that in the training data. Many algorithms and user interfaces have been designed to optimize and facilitate this human--machine interaction; however, fewer studies have addressed the potential defects the designs can cause. Effective interaction often requires exposing the user to the training data or its statistics. The design of the system is then critical, as this can lead to double use of data and overfitting, if the user reinforces noisy patterns in the data. We propose a user modelling methodology, by assuming simple rational behaviour, to correct the problem. We show, in a user study with 48 participants, that the method improves predictive performance in a sparse linear regression sentiment analysis task, where graded user knowledge on feature relevance is elicited. We believe that the key idea of inferring user knowledge with probabilistic user models has general applicability in guarding against overfitting and improving interactive machine learning.

Description

Keywords

Interactive machine learning, Probabilistic modeling, Bayesian Inference, overfitting, expert prior elicitation, human-in-the-loop machine learning

Other note

DOI

10.1145/3172944.3172989

Citation

Daee, P, Peltola, T, Vehtari, A & Kaski, S 2018, User Modelling for Avoiding Overfitting in Interactive Knowledge Elicitation for Prediction. in IUI 2018 - Proceedings of the 23rd International Conference on Intelligent User Interfaces. ACM, pp. 305-310, International Conference on Intelligent User Interfaces, Tokyo, Japan, 07/03/2018. https://doi.org/10.1145/3172944.3172989

Permanent link to this item

https://urn.fi/URN:NBN:fi:aalto-201812106382

Collections

[article-cris] Perustieteiden korkeakoulu / SCI