MaxInfoRL：情報利得最大化による強化学習の探索促進
MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximization
情報利得を内在的報酬として最大化し、価値関数と状態・報酬・行動のエントロピーを自然にトレードオフする探索フレームワークを提案。連続状態行動空間のオフポリシーRLに適用し、困難な探索問題や視覚制御タスクで性能向上を実現。
著者: Bhavya Sukhija, Stelian Coros, Andreas Krause, Pieter Abbeel, Carmelo Sferrazza
分類: cs.LG, cs.AI, cs.RO
原文アブストラクト
Reinforcement learning (RL) algorithms aim to balance exploiting the current best strategy with exploring new options that could lead to higher rewards. Most common RL algorithms use undirected exploration, i.e., select random sequences of actions. Exploration can also be directed using intrinsic rewards, such as curiosity or model epistemic uncertainty. However, effectively balancing task and intrinsic rewards is challenging and often task-dependent. In this work, we introduce a framework, MaxInfoRL, for balancing intrinsic and extrinsic exploration. MaxInfoRL steers exploration towards informative transitions, by maximizing intrinsic rewards such as the information gain about the underlying task. When combined with Boltzmann exploration, this approach naturally trades off maximization of the value function with that of the entropy over states, rewards, and actions. We show that our approach achieves sublinear regret in the simplified setting of multi-armed bandits. We then apply this general formulation to a variety of off-policy model-free RL methods for continuous state-action spaces, yielding novel algorithms that achieve superior performance across hard exploration problems and complex scenarios such as visual control tasks.
