Interactive Groupwise Comparison for Faster Reinforcement Learning from Human Feedback

Thumbnail Image

Files

master_Kompatscher_Jan_2025.pdf (3.39 MB)

URL

Journal Title

Journal ISSN

Volume Title

School of Science | Master's thesis
Unless otherwise stated, all rights belong to the author. You may download, display and print this publication for Your own personal use. Commercial use is prohibited.

Authors

Date

2024-12-09

Department

Major/Subject

Human-Computer Interaction and Design

Mcode

Degree programme

Master's Programme in ICT Innovation

Language

en

Pages

97

Series

Abstract

Reinforcement Learning from Human Feedback (RLHF) has gained significant attention due to its ability to align AI models with human preferences without designing complicated reward functions manually. However, the traditional RLHF approach via pairwise comparisons is labor-intensive and costly. As tasks become more complex, the amount of required feedback increases, making it difficult to scale. This work presents an interactive groupwise comparison approach for RLHF that exploits human expertise in comprising groups of similar behaviors. To support this concept, the user interface of our approach visualizes contextual pieces of information in a single exploration view, including hierarchical clustering of behaviors and human preferences, and using a hierarchical radial chart with edge bundling to avoid visual clutter. We built a visualization interface comprising two interactively linked views: 1) an exploration view showing a contextual overview of all sampled behaviors in a hierarchical clustering structure, and 2) a comparison view displaying two selected groups of behaviors for user queries. Users can efficiently explore large sets of behaviors by iterating between these two views. We evaluated the effectiveness and efficiency of the proposed approach compared with the traditional approach of pairwise comparisons using a simulated user model and a controlled user study with participants. Using groupwise comparisons, one can increase the number of elicited preferences within the same amount of human time by 71.2% with a lower error rate and obtain a better policy.

Description

Supervisor

Nieminen, Mika P.

Thesis advisor

Shi, Danqing
Varni, Giovanna

Keywords

machine learning, reinforcement learning, preference learning, visualization, UI design, active learning

Other note

Citation

Permanent link to this item

https://urn.fi/URN:NBN:fi:aalto-202501271897

Collections

[dipl] Perustieteiden korkeakoulu / SCI