終端状態での意図的な価値過小評価によるTD学習の報酬設計ミス対策
Intentionally-underestimated Value Function at Terminal State for Temporal-difference Learning with Mis-designed Reward
強化学習のエピソード終了時にTD学習が行う例外処理が、報酬設計によって意図しない過大評価を引き起こし誤った方策を学習する問題を解決するため、終端後の価値を意図的に過小評価する手法を提案し、実機実験で有効性を示した。
分類: cs.RO, cs.LG
原文アブストラクト
Robot control using reinforcement learning has become popular, but its learning process generally terminates halfway through an episode for safety and time-saving reasons. This study addresses the problem of the most popular exception handling that temporal-difference (TD) learning performs at such termination. That is, by forcibly assuming zero value after termination, unintentionally implicit underestimation or overestimation occurs, depending on the reward design in the normal states. When the episode is terminated due to task failure, the failure may be highly valued with the unintentional overestimation, and the wrong policy may be acquired. Although this problem can be avoided by paying attention to the reward design, it is essential in practical use of TD learning to review the exception handling at termination. This paper therefore proposes a method to intentionally underestimate the value after termination to avoid learning failures due to the unintentional overestimation. In addition, the degree of underestimation is adjusted according to the degree of stationarity at termination, thereby preventing excessive exploration due to the intentional underestimation. Simulations and real robot experiments showed that the proposed method can stably obtain the optimal policies for various tasks and reward designs. https://youtu.be/AxXr8uFOe7M
