Learning Mental States from Biosignals

Abstract

As computing technology evolves, users perform more complex tasks with computers. Hence, users expect from user interfaces to be more proactive than reactive. A proactive interface should anticipate the user’s intentions and take the right action without requiring a user command. The crucial first step for such an interface is to infer the user’s mental state, which gives important cues about user intentions. This thesis consists of several case studies on inferring mental states of computer users. Biosensing technology provides a variety of hardware tools for measuring several aspects of human physiology, which is correlated with emotions and mental processes. However, signals gathered with biosensors are notoriously noisy. The mainstream approach to overcome this noise is either to increase the signal precision by expensive and stationary sensors or to control the experiment setups more heavily. Both of these solutions undermine the usability of the developed methods in real-life user interfaces. In this thesis, machine learning is used as an alternative strategy for handling the biosignal noise in mental state inference. Computer users have been monitored under loosely controlled experiment setups by cheap and inaccurate biosensors, and novel machine learning models that infer mental states such as affective state, mental workload, relevance of a real-world object, and auditory attention are built. The methodological contributions of the thesis are mainly on multi-view learning and multitask learning. Multi-view learning is used for integrating signals of multiple biosensors and the stimuli. Multitask learning is used for inferring multiple mental states at once, and for exploiting the inter-subject similarities for higher prediction accuracy. A novel multitask learning algorithm that transfers knowledge across multi-view learning tasks is introduced. Another novelty is a Bayesian factor analyzer with a time-dependent latent space that captures the dynamic nature of biosignals better than methods that assume independent samples. The overall outcome of the thesis is that it is feasible to predict mental states from unobtrusive biosensors with reasonable accuracy using state-of-the-art machine learning models.