不確実性推定によるサンプル効率の高い深層強化学習
Sample Efficient Deep Reinforcement Learning via Uncertainty Estimation
強化学習におけるノイズの多い価値推定を、ベイズ的不確実性推定と逆分散重み付けで軽減し、サンプル効率を改善する手法を提案した論文。
著者: Vincent Mai, Kaustubh Mani, Liam Paull
分類: cs.LG, cs.AI, cs.RO
原文アブストラクト
In model-free deep reinforcement learning (RL) algorithms, using noisy value estimates to supervise policy evaluation and optimization is detrimental to the sample efficiency. As this noise is heteroscedastic, its effects can be mitigated using uncertainty-based weights in the optimization process. Previous methods rely on sampled ensembles, which do not capture all aspects of uncertainty. We provide a systematic analysis of the sources of uncertainty in the noisy supervision that occurs in RL, and introduce inverse-variance RL, a Bayesian framework which combines probabilistic ensembles and Batch Inverse Variance weighting. We propose a method whereby two complementary uncertainty estimation methods account for both the Q-value and the environment stochasticity to better mitigate the negative impacts of noisy supervision. Our results show significant improvement in terms of sample efficiency on discrete and continuous control tasks.
