状態拡張による人間の選好を用いた強化学習
A State Augmentation based approach to Reinforcement Learning from Human Preferences
人間の選好フィードバックから報酬モデルを学習する際、状態拡張により不変性を導入して性能を向上させる手法を提案し、複数タスクで有効性を示した。
著者: Mudit Verma, Subbarao Kambhampati
分類: cs.AI, cs.LG
原文アブストラクト
Reinforcement Learning has suffered from poor reward specification, and issues for reward hacking even in simple enough domains. Preference Based Reinforcement Learning attempts to solve the issue by utilizing binary feedbacks on queried trajectory pairs by a human in the loop indicating their preferences about the agent's behavior to learn a reward model. In this work, we present a state augmentation technique that allows the agent's reward model to be robust and follow an invariance consistency that significantly improved performance, i.e. the reward recovery and subsequent return computed using the learned policy over our baseline PEBBLE. We validate our method on three domains, Mountain Car, a locomotion task of Quadruped-Walk, and a robotic manipulation task of Sweep-Into, and find that using the proposed augmentation the agent not only benefits in the overall performance but does so, quite early in the agent's training phase.
