Multimodal Humor Detection and Social Perception Prediction
Loading...
Access rights
openAccess
CC BY
CC BY
publishedVersion
URL
Journal Title
Journal ISSN
Volume Title
A4 Artikkeli konferenssijulkaisussa
This publication is imported from Aalto University research portal.
View publication in the Research portal (opens in new window)
View/Open full text file from the Research portal (opens in new window)
Other link related to publication (opens in new window)
View publication in the Research portal (opens in new window)
View/Open full text file from the Research portal (opens in new window)
Other link related to publication (opens in new window)
Date
Major/Subject
Mcode
Degree programme
Language
en
Pages
5
Series
MuSe 2024 - Proceedings of the 5th Multimodal Sentiment Analysis Challenge and Workshop: Social Perception and Humor, Co-Located with: MM 2024, pp. 60-64
Abstract
The parallel audio-visual-text data contains vast amount of information. Thus it is essential to develop machine learning algorithms that can utilise them efficiently. In this work, we investigated unimodal and multimodal solutions for MuSe Humor and Perception challenges. Our main goal was to explicitly show the contribution of each modality in the multimodal systems. In addition, for the Humor challenge, we examined the effect of extending the input context and smoothing the framewise predictions. For Perception challenge, we trained an attention-encoder-decoder model to predict all perceived labels with a single model. During the challenge, the best results were achieved by a fusion of unimodal and multimodal systems, AUC = 0.8645 for Humor, and mean Pearson’s correlation ρ = 0.3550 for Perception. By investigating the multimodal systems we found that using only part of the video for model training can be beneficial, suggesting that valuable information is condensed to certain parts of the video. The implementation of our models and experiments can be found at https://github.com/aalto-speech/MuSe-2024.Description
Publisher Copyright: © 2024 Copyright held by the owner/author(s).
Other note
Citation
Bijoy, M H, Porjazovski, D, Phan, N, Huang, G, Grósz, T & Kurimo, M 2024, Multimodal Humor Detection and Social Perception Prediction. in MuSe 2024 - Proceedings of the 5th Multimodal Sentiment Analysis Challenge and Workshop : Social Perception and Humor, Co-Located with: MM 2024. MuSe 2024 - Proceedings of the 5th Multimodal Sentiment Analysis Challenge and Workshop: Social Perception and Humor, Co-Located with: MM 2024, ACM, pp. 60-64, Multimodal Sentiment Analysis Challenge and Workshop, Melbourne, Australia, 28/10/2024. https://doi.org/10.1145/3689062.3689376