Proactive robot task sequencing through real-time hand motion prediction in human–robot collaboration
No Thumbnail Available
Access rights
openAccess
CC BY
CC BY
publishedVersion
URL
Journal Title
Journal ISSN
Volume Title
A1 Alkuperäisartikkeli tieteellisessä aikakauslehdessä
This publication is imported from Aalto University research portal.
View publication in the Research portal (opens in new window)
View/Open full text file from the Research portal (opens in new window)
Other link related to publication (opens in new window)
View publication in the Research portal (opens in new window)
View/Open full text file from the Research portal (opens in new window)
Other link related to publication (opens in new window)
Unless otherwise stated, all rights belong to the author. You may download, display and print this publication for Your own personal use. Commercial use is prohibited.
Date
2025-03
Major/Subject
Mcode
Degree programme
Language
en
Pages
11
Series
Image and Vision Computing, Volume 155
Abstract
Human–robot collaboration (HRC) is essential for improving productivity and safety across various industries. While reactive motion re-planning strategies are useful, there is a growing demand for proactive methods that predict human intentions to enable more efficient collaboration. This study addresses this need by introducing a framework that combines deep learning-based human hand trajectory forecasting with heuristic optimization for robotic task sequencing. The deep learning model advances real-time hand position forecasting using a multi-task learning loss to account for both hand positions and contact delay regression, achieving state-of-the-art performance on the Ego4D Future Hand Prediction benchmark. By integrating hand trajectory predictions into task planning, the framework offers a cohesive solution for HRC. To optimize task sequencing, the framework incorporates a Dynamic Variable Neighborhood Search (DynamicVNS) heuristic algorithm, which allows robots to pre-plan task sequences and avoid potential collisions with human hand positions. DynamicVNS provides significant computational advantages over the generalized VNS method. The framework was validated on a UR10e robot performing a visual inspection task in a HRC scenario, where the robot effectively anticipated and responded to human hand movements in a shared workspace. Experimental results highlight the system's effectiveness and potential to enhance HRC in industrial settings by combining predictive accuracy and task planning efficiency.Description
Publisher Copyright: © 2025
Keywords
Egocentric vision, Human–robot collaboration
Other note
Citation
Abilkassov, S, Gentner, M, Shintemirov, A, Steinbach, E & Popa, M 2025, ' Proactive robot task sequencing through real-time hand motion prediction in human–robot collaboration ', Image and Vision Computing, vol. 155, 105443 . https://doi.org/10.1016/j.imavis.2025.105443