FLoRA: 低ランクスタイル適応によるサンプル効率の良い選好ベース強化学習
FLoRA: Sample-Efficient Preference-based RL via Low-Rank Style Adaptation of Reward Functions
事前学習済みロボットの報酬モデルに低ランク行列を追加して選好に適応させることで、元タスクの性能を保ちながら少ない選好データでスタイル適応を実現する手法を提案。
著者: Daniel Marta, Simon Holk, Miguel Vasco, Jens Lundell, Timon Homberger, Finn Busch, Olov Andersson, Danica Kragic, Iolanda Leite
分類: cs.RO, cs.LG
原文アブストラクト
Preference-based reinforcement learning (PbRL) is a suitable approach for style adaptation of pre-trained robotic behavior: adapting the robot's policy to follow human user preferences while still being able to perform the original task. However, collecting preferences for the adaptation process in robotics is often challenging and time-consuming. In this work we explore the adaptation of pre-trained robots in the low-preference-data regime. We show that, in this regime, recent adaptation approaches suffer from catastrophic reward forgetting (CRF), where the updated reward model overfits to the new preferences, leading the agent to become unable to perform the original task. To mitigate CRF, we propose to enhance the original reward model with a small number of parameters (low-rank matrices) responsible for modeling the preference adaptation. Our evaluation shows that our method can efficiently and effectively adjust robotic behavior to human preferences across simulation benchmark tasks and multiple real-world robotic tasks.
