Residual Learning and Context Encoding for Adaptive Offline-to-Online Reinforcement Learning

Offline reinforcement learning (RL) allows learning sequential behavior from fixed datasets. Since offline datasets do not cover all possible situations, many methods collect additional data during online fine-tuning to improve performance. In general, these methods assume that the transition dynamics remain the same during both the offline and online phases of training. However, in many real-world applications, such as outdoor construction and navigation over rough terrain, it is common for the transition dynamics to vary between the offline and online phases. Moreover, the dynamics may vary during the online fine-tuning. To address this problem of changing dynamics from offline to online RL we propose a residual learning approach that infers dynamics changes to correct the outputs of the offline solution. At the online fine-tuning phase, we train a context encoder to learn a representation that is consistent inside the current online learning environment while being able to predict dynamic transitions.

Description

Keywords

Adaptive RL, Context Encoding, Offline-to-Online RL

Citation

Nakhaeinezhadfard, M, Scannell, A & Pajarinen, J 2024, 'Residual Learning and Context Encoding for Adaptive Offline-to-Online Reinforcement Learning', Proceedings of Machine Learning Research, vol. 242, pp. 1107-1121. < https://proceedings.mlr.press/v242/nakhaeinezhadfard24a.html >

Permanent link to this item

https://urn.fi/URN:NBN:fi:aalto-202410026594

Collections

[article-cris] Sähkötekniikan korkeakoulu / ELEC

Show all metadata

Residual Learning and Context Encoding for Adaptive Offline-to-Online Reinforcement Learning

Access rights

URL

Journal Title

Journal ISSN

Volume Title

Authors

Date

Department

Major/Subject

Mcode

Degree programme

Language

Pages

Series

Abstract

Description

Keywords

Other note

Citation

Permanent link to this item

Collections