ソフト分解方策-批評家：離散RLによる連続制御のギャップを埋める
Soft Decomposed Policy-Critic: Bridging the Gap for Effective Continuous Control with Discrete RL
連続制御問題に対して、各行動次元を独立に離散化し、ソフトQ関数を最大化する共有批評家ネットワークを用いるSDPCアーキテクチャを提案し、MujocoやBox2dのタスクで既存の連続RL手法を上回る性能を示した。
著者: Yechen Zhang, Jian Sun, Gang Wang, Zhuo Li, Wei Chen
分類: cs.LG, cs.AI
原文アブストラクト
Discrete reinforcement learning (RL) algorithms have demonstrated exceptional performance in solving sequential decision tasks with discrete action spaces, such as Atari games. However, their effectiveness is hindered when applied to continuous control problems due to the challenge of dimensional explosion. In this paper, we present the Soft Decomposed Policy-Critic (SDPC) architecture, which combines soft RL and actor-critic techniques with discrete RL methods to overcome this limitation. SDPC discretizes each action dimension independently and employs a shared critic network to maximize the soft $Q$-function. This novel approach enables SDPC to support two types of policies: decomposed actors that lead to the Soft Decomposed Actor-Critic (SDAC) algorithm, and decomposed $Q$-networks that generate Boltzmann soft exploration policies, resulting in the Soft Decomposed-Critic Q (SDCQ) algorithm. Through extensive experiments, we demonstrate that our proposed approach outperforms state-of-the-art continuous RL algorithms in a variety of continuous control tasks, including Mujoco's Humanoid and Box2d's BipedalWalker. These empirical results validate the effectiveness of the SDPC architecture in addressing the challenges associated with continuous control.
