強化学習における平均報酬と割引報酬の最適性基準の再検討
Examining average and discounted reward optimality criteria in reinforcement learning
強化学習の最適性基準である平均報酬と割引報酬の関係を体系的に整理し、人工的な割引因子の問題点と割引不要な平均報酬基準を直接最大化する利点を論じた論文。
著者: Vektor Dewanto, Marcus Gallagher
分類: cs.LG, cs.AI, cs.RO, cs.SY, eess.SY
原文アブストラクト
In reinforcement learning (RL), the goal is to obtain an optimal policy, for which the optimality criterion is fundamentally important. Two major optimality criteria are average and discounted rewards. While the latter is more popular, it is problematic to apply in environments without an inherent notion of discounting. This motivates us to revisit a) the progression of optimality criteria in dynamic programming, b) justification for and complication of an artificial discount factor, and c) benefits of directly maximizing the average reward criterion, which is discounting-free. Our contributions include a thorough examination of the relationship between average and discounted rewards, as well as a discussion of their pros and cons in RL. We emphasize that average-reward RL methods possess the ingredient and mechanism for applying a family of discounting-free optimality criteria (Veinott, 1969) to RL.
