Computational representations for user interfaces

Thumbnail Image

Files

isbn9789526427966.pdf (69.72 MB)

URL

Journal Title

Journal ISSN

Volume Title

School of Electrical Engineering | Doctoral thesis (article-based) | Defence date: 2025-10-17
Unless otherwise stated, all rights belong to the author. You may download, display and print this publication for Your own personal use. Commercial use is prohibited.

Authors

Date

2025

Department

Informaatio- ja tietoliikennetekniikan laitos
Department of Information and Communications Engineering

Major/Subject

Mcode

Degree programme

Language

en

Pages

175 + app. 81

Series

Aalto University publication series Doctoral Theses, 212/2025

Abstract

Traditional graphical user interfaces (GUIs) often follow a “one-size-fits-all” design, failing to accommodate the diverse needs and contexts of all their users. What if interfaces could instead dynamically understand and adapt to individual users, enhancing their capabilities across a wide range of tasks and contexts? As the diversity of user needs expands, the challenge of designing GUIs that accommodate varying contexts becomes increasingly complex. Data-driven AI methods may offer a way to design GUIs that align with users’ goals. Current AI methods, however, often fall short of capturing the full complexity of human needs, particularly when considering domain-specific knowledge and user-specific requirements. This dissertation contributes to the development of human-centered neural representations for interactions that combine domain knowledge and data-driven learning for GUIs. This dissertation centers on the development of intelligent GUIs in two primary areas. First, we focus on creating computational representations that capture the essential properties of UI design. Specifically, these representations integrate domain-specific knowledge into AI models, allowing for design expert guidance while ensuring that users retain control over their interactions. Moreover, we develop AI models that simulate and predict human behaviors to facilitate automatic personalized adaptation. These models encompass various human behaviors, including eye movements and user interactions. Simulating these behaviors enables personalized optimization and enhances the AI to adaptively respond to user needs.

Description

Supervising professor

Oulasvirta, Antti, Prof., Aalto University, Department of Information and Communications Engineering, Finland

Thesis advisor

Oulasvirta, Antti, Prof., Aalto University, Department of Information and Communications Engineering, Finland
Garg, Vikas, Asst. Prof., Aalto University, Department of Computer Science, Finland

Keywords

computational representation, user interface, eye tracking

Other note

Parts

  • [Publication 1]: Yue Jiang, Changkong Zhou, Vikas Garg, Antti Oulasvirta. Graph4GUI: Graph Neural Networks for Representing Graphical User Interfaces. In Proceedings of the 42nd Annual SIGCHI Conference on Human Factors in Computing Systems (CHI2024), May 2024.
    Full text in Acris/Aaltodoc: https://urn.fi/URN:NBN:fi:aalto-202405223492
    DOI: 10.1145/3613904.3642822 View at publisher
  • [Publication 2]: Yue Jiang, Luis A. Leiva, Hamed Rezazadegan Tavakoli, Paul R. B. Houssel, Julia Kylmala, Antti Oulasvirta. UEyes: Understanding Visual Saliency across User Interface Types. In Proceedings of the 41st Annual SIGCHI Conference on Human Factors in Computing Systems (CHI2023), May 2023.
    Full text in Acris/Aaltodoc: https://urn.fi/URN:NBN:fi:aalto-202306304372
    DOI: 10.1145/3544548.3581096 View at publisher
  • [Publication 3]: Yue Jiang*, Zixin Guo*, Hamed Rezazadegan Tavakoli, Luis A. Leiva, Antti Oulasvirta. EyeFormer: Predicting Personalized Scanpaths with Transformer-Guided Reinforcement Learning. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (UIST2024), October 2024.
    Full text in Acris/Aaltodoc: https://urn.fi/URN:NBN:fi:aalto-202501101102
    DOI: 10.1145/3654777.3676436 View at publisher
  • [Publication 4]: Yue Jiang, Eldon Schoop, Amanda Swearngin, Jeffrey Nichols. ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations. In Proceedings of the 30th Annual ACM Conference on Intelligent User Interfaces (IUI2025), March 2025.
    Full text in Acris/Aaltodoc: https://urn.fi/URN:NBN:fi:aalto-202504163256
    DOI: 10.1145/3708359.3712129 View at publisher

Citation

Permanent link to this item

https://urn.fi/URN:ISBN:978-952-64-2796-6

Collections

[diss] Sähkötekniikan korkeakoulu / ELEC