大規模離散行動空間のための確率的Q学習
Stochastic Q-learning for Large Discrete Action Spaces
大規模離散行動空間において、全行動ではなく対数オーダーの確率的な行動部分集合のみを考慮することで、Q学習やDQNの計算負荷を大幅に削減する手法を提案し、収束性の理論解析と実験的検証を行った。
著者: Fares Fourati, Vaneet Aggarwal, Mohamed-Slim Alouini
分類: cs.LG, cs.AI, cs.PF, cs.RO, stat.ML
原文アブストラクト
In complex environments with large discrete action spaces, effective decision-making is critical in reinforcement learning (RL). Despite the widespread use of value-based RL approaches like Q-learning, they come with a computational burden, necessitating the maximization of a value function over all actions in each iteration. This burden becomes particularly challenging when addressing large-scale problems and using deep neural networks as function approximators. In this paper, we present stochastic value-based RL approaches which, in each iteration, as opposed to optimizing over the entire set of $n$ actions, only consider a variable stochastic set of a sublinear number of actions, possibly as small as $\mathcal{O}(\log(n))$. The presented stochastic value-based RL methods include, among others, Stochastic Q-learning, StochDQN, and StochDDQN, all of which integrate this stochastic approach for both value-function updates and action selection. The theoretical convergence of Stochastic Q-learning is established, while an analysis of stochastic maximization is provided. Moreover, through empirical validation, we illustrate that the various proposed approaches outperform the baseline methods across diverse environments, including different control problems, achieving near-optimal average returns in significantly reduced time.
