AS-Net : active speaker detection using deep audio-visual attention
No Thumbnail Available
Access rights
openAccess
URL
Journal Title
Journal ISSN
Volume Title
A1 Alkuperäisartikkeli tieteellisessä aikakauslehdessä
This publication is imported from Aalto University research portal.
View publication in the Research portal (opens in new window)
View/Open full text file from the Research portal (opens in new window)
Other link related to publication (opens in new window)
View publication in the Research portal (opens in new window)
View/Open full text file from the Research portal (opens in new window)
Other link related to publication (opens in new window)
Authors
Date
2024-08
Department
Major/Subject
Mcode
Degree programme
Language
en
Pages
16
Series
Multimedia Tools and Applications, Volume 83, issue 28, pp. 72027-72042
Abstract
Active Speaker Detection (ASD) aims at identifying the active speaker among multiple speakers in a video scene. Previous ASD models often seek audio and visual features from long video clips with a complex 3D Convolutional Neural Network (CNN) architecture. However, models based on 3D CNNs can generate discriminative spatial-temporal features, but this comes at the expense of computational complexity, and they frequently face challenges in detecting active speakers in short video clips. This work proposes the Active Speaker Network (AS-Net) model, a simple yet effective ASD method tailored for detecting active speakers in relatively short video clips without relying on 3D CNNs. Instead, it incorporates the Temporal Shift Module (TSM) into 2D CNNs, facilitating the extraction of dense temporal visual features without the need for additional computations. Moreover, self-attention and cross-attention schemes are introduced to enhance long-term temporal audio-visual synchronization, thereby improving ASD performance. Experimental results demonstrate that AS-Net outperforms state-of-the-art 2D CNN-based methods on the AVA-ActiveSpeaker dataset and remains competitive with the methods utilizing more complex architectures.Description
Funding Information: This work is supported by the Academy of Finland in project 345791. We acknowledge the LUMI supercomputer, owned by the EuroHPC Joint Undertaking, hosted by CSC and the LUMI consortium. Publisher Copyright: © 2024, The Author(s).
Keywords
Active speaker detection, Audio-visual attention, Audio-visual features, Convolutional Neural Networks (CNNs), Temporal shift module
Other note
Citation
Radman, A & Laaksonen, J 2024, ' AS-Net : active speaker detection using deep audio-visual attention ', Multimedia Tools and Applications, vol. 83, no. 28, pp. 72027-72042 . https://doi.org/10.1007/s11042-024-18457-9