EyeFormer: Predicting Personalized Scanpaths with Transformer-Guided Reinforcement Learning

From a visual-perception perspective, modern graphical user interfaces (GUIs) comprise a complex graphics-rich two-dimensional visuospatial arrangement of text, images, and interactive objects such as buttons and menus. While existing models can accurately predict regions and objects that are likely to attract attention “on average”, no scanpath model has been capable of predicting scanpaths for an individual. To close this gap, we introduce EyeFormer, which utilizes a Transformer architecture as a policy network to guide a deep reinforcement learning algorithm that predicts gaze locations. Our model offers the unique capability of producing personalized predictions when given a few user scanpath samples. It can predict full scanpath information, including fixation positions and durations, across individuals and various stimulus types. Additionally, we demonstrate applications in GUI layout optimization driven by our model.

DOI

10.1145/3654777.3676436

Citation

Jiang, Y, Guo, Z, Rezazadegan Tavakoli, H, Leiva, L A & Oulasvirta, A 2024, EyeFormer: Predicting Personalized Scanpaths with Transformer-Guided Reinforcement Learning. in Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology., 47, UIST '24, ACM, New York, ACM Symposium on User Interface Software and Technology, Pittsburgh, Pennsylvania, United States, 13/10/2024. https://doi.org/10.1145/3654777.3676436

Permanent link to this item

https://urn.fi/URN:NBN:fi:aalto-202501101102

Collections

[article-cris] Perustieteiden korkeakoulu / SCI

Show all metadata

EyeFormer: Predicting Personalized Scanpaths with Transformer-Guided Reinforcement Learning

Access rights

URL

Journal Title

Journal ISSN

Volume Title

Authors

Date

Department

Major/Subject

Mcode

Degree programme

Language

Pages

Series

Abstract

Description

Keywords

Other note

DOI

Citation

Permanent link to this item

Collections