Temporal teacher with masked transformers for semi-supervised action proposal generation

Loading...
Thumbnail Image
Journal Title
Journal ISSN
Volume Title
A1 Alkuperäisartikkeli tieteellisessä aikakauslehdessä
Date
2024-03-15
Major/Subject
Mcode
Degree programme
Language
en
Pages
15
Series
MACHINE VISION AND APPLICATIONS, Volume 35, issue 3, pp. 1-15
Abstract
By conditioning on unit-level predictions, anchor-free models for action proposal generation have displayed impressive capabilities, such as having a lightweight architecture. However, task performance depends significantly on the quality of data used in training, and most effective models have relied on human-annotated data. Semi-supervised learning, i.e., jointly training deep neural networks with a labeled dataset as well as an unlabeled dataset, has made significant progress recently. Existing works have either primarily focused on classification tasks, which may require less annotation effort, or considered anchor-based detection models. Inspired by recent advances in semi-supervised methods on anchor-free object detectors, we propose a teacher-student framework for a two-stage action detection pipeline, named Temporal Teacher with Masked Transformers (TTMT), to generate high-quality action proposals based on an anchor-free transformer model. Leveraging consistency learning as one self-training technique, the model jointly trains an anchor-free student model and a gradually progressing teacher counterpart in a mutually beneficial manner. As the core model, we design a Transformer-based anchor-free model to improve effectiveness for temporal evaluation. We integrate bi-directional masks and devise encoder-only Masked Transformers for sequences. Jointly training on boundary locations and various local snippet-based features, our model predicts via the proposed scoring function for generating proposal candidates. Experiments on the THUMOS14 and ActivityNet-1.3 benchmarks demonstrate the effectiveness of our model for temporal proposal generation task.
Description
Publisher Copyright: © The Author(s) 2024.
Keywords
Anchor-free model, Semi-supervised learning, Temporal proposal generation, Transformer network
Other note
Citation
Pehlivan, S & Laaksonen, J 2024, ' Temporal teacher with masked transformers for semi-supervised action proposal generation ', MACHINE VISION AND APPLICATIONS, vol. 35, no. 3, 36, pp. 1-15 . https://doi.org/10.1007/s00138-024-01521-7