D-Shape: 目標条件付けによる模倣形状強化学習
D-Shape: Demonstration-Shaped Reinforcement Learning via Goal Conditioning
模倣学習と強化学習を報酬整形と目標条件付きRLで統合し、劣ったデモからでも最適方策を学習できる手法を提案。
著者: Caroline Wang, Garrett Warnell, Peter Stone
分類: cs.LG, cs.AI, cs.HC, cs.RO
原文アブストラクト
While combining imitation learning (IL) and reinforcement learning (RL) is a promising way to address poor sample efficiency in autonomous behavior acquisition, methods that do so typically assume that the requisite behavior demonstrations are provided by an expert that behaves optimally with respect to a task reward. If, however, suboptimal demonstrations are provided, a fundamental challenge appears in that the demonstration-matching objective of IL conflicts with the return-maximization objective of RL. This paper introduces D-Shape, a new method for combining IL and RL that uses ideas from reward shaping and goal-conditioned RL to resolve the above conflict. D-Shape allows learning from suboptimal demonstrations while retaining the ability to find the optimal policy with respect to the task reward. We experimentally validate D-Shape in sparse-reward gridworld domains, showing that it both improves over RL in terms of sample efficiency and converges consistently to the optimal policy in the presence of suboptimal demonstrations.
