Development of a speech emotion recognizer for large-scale child-centered audio recordings from a hospital environment

Thumbnail Image
Journal Title
Journal ISSN
Volume Title
A1 Alkuperäisartikkeli tieteellisessä aikakauslehdessä
Degree programme
Speech Communication, Volume 148
In order to study how early emotional experiences shape infant development, one approach is to analyze the emotional content of speech heard by infants, as captured by child-centered daylong recordings, and as analyzed by automatic speech emotion recognition (SER) systems. However, since large-scale daylong audio is initially unannotated and differs from typical speech corpora from controlled environments, there are no existing in-domain SER systems for the task. Based on existing literature, it is also unclear what is the best approach to deploy a SER system for a new domain. Consequently, in this study, we investigated alternative strategies for deploying a SER system for large-scale child-centered audio recordings from a neonatal hospital environment, comparing cross-corpus generalization, active learning (AL), and domain adaptation (DA) methods in the process. We first conducted simulations with existing emotion-labeled speech corpora to find the best strategy for SER system deployment. We then tested howthe findings generalize to our new initially unannotated dataset. As a result, we found that the studied AL method provided overall the most consistent results, being less dependent on the specifics of the training corpora or speech features compared to the alternative methods. However, in situations without the possibility to annotate data, unsupervised DA proved to be the best approach. We also observed that deployment of a SER system for real-world daylong child-centered audio recordings achieved a SER performance level comparable to those reported in literature, and that the amount of human effort required for the system deployment was overall relatively modest.
Funding Information: This research was funded by Academy of Finland grants no. 314573 , 314602 , 332962 , and 335872 , and EU Horizon-2020 grant no. 957337 MARVEL. The authors would like to thank the APPLE consortium for the help in the project. Publisher Copyright: © 2023 The Author(s) | openaire: EC/H2020/957337/EU//MARVEL
Daylong audio, LENA recorder, Real-world audio, Speech analysis, Speech emotion recognition
Other note
Vaaras , E , Ahlqvist-Björkroth , S , Drossos , K , Lehtonen , L & Räsänen , O 2023 , ' Development of a speech emotion recognizer for large-scale child-centered audio recordings from a hospital environment ' , Speech Communication , vol. 148 , pp. 9-22 .