Multimodal Humor Detection and Social Perception Prediction

Abstract

The parallel audio-visual-text data contains vast amount of information. Thus it is essential to develop machine learning algorithms that can utilise them efficiently. In this work, we investigated unimodal and multimodal solutions for MuSe Humor and Perception challenges. Our main goal was to explicitly show the contribution of each modality in the multimodal systems. In addition, for the Humor challenge, we examined the effect of extending the input context and smoothing the framewise predictions. For Perception challenge, we trained an attention-encoder-decoder model to predict all perceived labels with a single model. During the challenge, the best results were achieved by a fusion of unimodal and multimodal systems, AUC = 0.8645 for Humor, and mean Pearson’s correlation ρ = 0.3550 for Perception. By investigating the multimodal systems we found that using only part of the video for model training can be beneficial, suggesting that valuable information is condensed to certain parts of the video. The implementation of our models and experiments can be found at https://github.com/aalto-speech/MuSe-2024.