自己教師ありオンライン報酬整形による疎報酬環境の効率化
Self-Supervised Online Reward Shaping in Sparse-Reward Environments
疎な報酬しか得られない環境で、軌跡のランキングを自己教師信号として報酬関数を自動的に密化し、任意のRLアルゴリズムのサンプル効率を向上させる手法を提案した。
著者: Farzan Memarian, Wonjoon Goo, Rudolf Lioutikov, Scott Niekum, Ufuk Topcu
分類: cs.LG, cs.RO
原文アブストラクト
We introduce Self-supervised Online Reward Shaping (SORS), which aims to improve the sample efficiency of any RL algorithm in sparse-reward environments by automatically densifying rewards. The proposed framework alternates between classification-based reward inference and policy update steps -- the original sparse reward provides a self-supervisory signal for reward inference by ranking trajectories that the agent observes, while the policy update is performed with the newly inferred, typically dense reward function. We introduce theory that shows that, under certain conditions, this alteration of the reward function will not change the optimal policy of the original MDP, while potentially increasing learning speed significantly. Experimental results on several sparse-reward environments demonstrate that, across multiple domains, the proposed algorithm is not only significantly more sample efficient than a standard RL baseline using sparse rewards, but, at times, also achieves similar sample efficiency compared to when hand-designed dense reward functions are used.
