Temporal modelling of first-person actions using hand-centric verb and object streams
No Thumbnail Available
Access rights
openAccess
acceptedVersion
URL
Journal Title
Journal ISSN
Volume Title
A1 Alkuperäisartikkeli tieteellisessä aikakauslehdessä
This publication is imported from Aalto University research portal.
View publication in the Research portal (opens in new window)
View/Open full text file from the Research portal (opens in new window)
Other link related to publication (opens in new window)
View publication in the Research portal (opens in new window)
View/Open full text file from the Research portal (opens in new window)
Other link related to publication (opens in new window)
Authors
Date
2021-11
Department
Major/Subject
Mcode
Degree programme
Language
en
Pages
17
Series
SIGNAL PROCESSING: IMAGE COMMUNICATION, Volume 99
Abstract
Analysis of first-person (egocentric) videos involving human actions could help in the solutions of many problems. These videos include a large number of fine-grained action categories with hand–object interactions. In this paper, a compositional verb–noun model including two complementary temporal streams is proposed with various fusion strategies to recognize egocentric actions. The first step is based on construction of verb and object video models as decomposition of actions with a special attention on hands. Particularly, the verb video model that is the spatial–temporal encoding of hand actions and the object video model that is the object scores with hand–object layout are represented as two separate pathways. The second step is the fusion stage to identify action category, where distinct verb and object models are combined to give their action judgments. We propose fusion strategies with recurrent steps collecting verb and object label judgments along a temporal video sequence. We evaluate recognition performances for individual verb and object models; and we present extensive experimental evaluations for action recognition over recurrent-based fusion approaches on the EGTEA Gaze+ dataset.Description
Publisher Copyright: © 2021 Elsevier B.V.
Keywords
Action recognition, Egocentric vision, First-person vision, RNN, Temporal models
Other note
Citation
Gökce, Z & Pehlivan, S 2021, ' Temporal modelling of first-person actions using hand-centric verb and object streams ', SIGNAL PROCESSING: IMAGE COMMUNICATION, vol. 99, 116436 . https://doi.org/10.1016/j.image.2021.116436