後知恵方策勾配法
Hindsight policy gradients
強化学習の方策勾配法に「後知恵」の仕組みを導入し、意図した目標とは別の目標達成度の情報を活用することで、疎な報酬環境でのサンプル効率を大幅に向上させた。
著者: Paulo Rauber, Avinash Ummadisingu, Filipe Mutz, Juergen Schmidhuber
分類: cs.LG, cs.AI, cs.NE, cs.RO
原文アブストラクト
A reinforcement learning agent that needs to pursue different goals across episodes requires a goal-conditional policy. In addition to their potential to generalize desirable behavior to unseen goals, such policies may also enable higher-level planning based on subgoals. In sparse-reward environments, the capacity to exploit information about the degree to which an arbitrary goal has been achieved while another goal was intended appears crucial to enable sample efficient learning. However, reinforcement learning agents have only recently been endowed with such capacity for hindsight. In this paper, we demonstrate how hindsight can be introduced to policy gradient methods, generalizing this idea to a broad class of successful algorithms. Our experiments on a diverse selection of sparse-reward environments show that hindsight leads to a remarkable increase in sample efficiency.
