Graph4GUI: Graph Neural Networks for Representing Graphical User Interfaces
Loading...
Access rights
openAccess
URL
Journal Title
Journal ISSN
Volume Title
A4 Artikkeli konferenssijulkaisussa
This publication is imported from Aalto University research portal.
View publication in the Research portal (opens in new window)
View/Open full text file from the Research portal (opens in new window)
Other link related to publication (opens in new window)
View publication in the Research portal (opens in new window)
View/Open full text file from the Research portal (opens in new window)
Other link related to publication (opens in new window)
Date
2024-05-11
Major/Subject
Mcode
Degree programme
Language
en
Pages
18
Series
Abstract
Present-day graphical user interfaces (GUIs) exhibit diverse arrangements of text, graphics, and interactive elements such as buttons and menus, but representations of GUIs have not kept up. They do not encapsulate both semantic and visuo-spatial relationships among elements. To seize machine learning’s potential for GUIs more efficiently, Graph4GUI exploits graph neural networks to capture individual elements’ properties and their semantic—visuo-spatial constraints in a layout. The learned representation demonstrated its effectiveness in multiple tasks, especially generating designs in a challenging GUI autocompletion task, which involved predicting the positions of remaining unplaced elements in a partially completed GUI. The new model’s suggestions showed alignment and visual appeal superior to the baseline method and received higher subjective ratings for preference. Furthermore, we demonstrate the practical benefits and efficiency advantages designers perceive when utilizing our model as an autocompletion plug-in.Description
Keywords
User Interface Representation, Constraint-based Layout, Graphical User Interface, Graph Neural Networks
Other note
Citation
Jiang, Y, Zhou, C, Garg, V & Oulasvirta, A 2024, Graph4GUI: Graph Neural Networks for Representing Graphical User Interfaces . in F F Mueller, P Kyburz, J R Williamson, C Sas, M L Wilson, P Toups Dugas & I Shklovski (eds), CHI '24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems ., 988, ACM, ACM SIGCHI Annual Conference on Human Factors in Computing Systems, Honolulu, Hawaii, United States, 11/05/2024 . https://doi.org/10.1145/3613904.3642822