AACHER: 後方視的経験再生を用いた多様なアクター・クリティック深層強化学習
AACHER: Assorted Actor-Critic Deep Reinforcement Learning with Hindsight Experience Replay
DDPGのアクターとクリティックを複数化して平均を取ることで安定性を高め、後方視的経験再生(HER)と組み合わせたAACHERを提案し、目標ベースのロボットタスクで性能を向上させた。
著者: Adarsh Sehgal, Muskan Sehgal, Hung Manh La
分類: cs.RO, cs.LG
原文アブストラクト
Actor learning and critic learning are two components of the outstanding and mostly used Deep Deterministic Policy Gradient (DDPG) reinforcement learning method. Since actor and critic learning plays a significant role in the overall robot's learning, the performance of the DDPG approach is relatively sensitive and unstable as a result. We propose a multi-actor-critic DDPG for reliable actor-critic learning to further enhance the performance and stability of DDPG. This multi-actor-critic DDPG is then integrated with Hindsight Experience Replay (HER) to form our new deep learning framework called AACHER. AACHER uses the average value of multiple actors or critics to substitute the single actor or critic in DDPG to increase resistance in the case when one actor or critic performs poorly. Numerous independent actors and critics can also gain knowledge from the environment more broadly. We implemented our proposed AACHER on goal-based environments: AuboReach, FetchReach-v1, FetchPush-v1, FetchSlide-v1, and FetchPickAndPlace-v1. For our experiments, we used various instances of actor/critic combinations, among which A10C10 and A20C20 were the best-performing combinations. Overall results show that AACHER outperforms the traditional algorithm (DDPG+HER) in all of the actor/critic number combinations that are used for evaluation. When used on FetchPickAndPlace-v1, the performance boost for A20C20 is as high as roughly 3.8 times the success rate in DDPG+HER.
