大規模行動空間における強化学習のためのアドバンテージベース最適化手法
An Advantage-based Optimization Method for Reinforcement Learning in Large Action Space
大規模行動空間での強化学習において、各次元の行動価値をアドバンテージで調整するABQを提案し、BDQや連続行動ベースラインを上回る性能を示した。
著者: Hai Lin, Cheng Huang, Zhihong Chen
分類: cs.LG, cs.AI
原文アブストラクト
Reinforcement learning tasks in real-world scenarios often involve large, high-dimensional action spaces, leading to challenges such as convergence difficulties, instability, and high computational complexity. It is widely acknowledged that traditional value-based reinforcement learning algorithms struggle to address these issues effectively. A prevalent approach involves generating independent sub-actions within each dimension of the action space. However, this method introduces bias, hindering the learning of optimal policies. In this paper, we propose an advantage-based optimization method and an algorithm named Advantage Branching Dueling Q-network (ABQ). ABQ incorporates a baseline mechanism to tune the action value of each dimension, leveraging the advantage relationship across different sub-actions. With this approach, the learned policy can be optimized for each dimension. Empirical results demonstrate that ABQ outperforms BDQ, achieving 3%, 171%, and 84% more cumulative rewards in HalfCheetah, Ant, and Humanoid environments, respectively. Furthermore, ABQ exhibits competitive performance when compared against two continuous action benchmark algorithms, DDPG and TD3.
