実機ロボットにおける同期型強化学習と非同期型強化学習の比較
Synchronous vs Asynchronous Reinforcement Learning in a Real World Robot
Franka Emika Pandaロボットアームを用いて、同期型と非同期型の強化学習を比較し、非同期型の方が学習が速く、より高いリターンを得られることを示した。
著者: Ali Parsaee, Fahim Shahriar, Chuxin He, Ruiqing Tan
分類: cs.RO, cs.AI, cs.CV, cs.LG
原文アブストラクト
In recent times, reinforcement learning (RL) with physical robots has attracted the attention of a wide range of researchers. However, state-of-the-art RL algorithms do not consider that physical environments do not wait for the RL agent to make decisions or updates. RL agents learn by periodically conducting computationally expensive gradient updates. When decision-making and gradient update tasks are carried out sequentially by the RL agent in a physical robot, it significantly increases the agent's response time. In a rapidly changing environment, this increased response time may be detrimental to the performance of the learning agent. Asynchronous RL methods, which separate the computation of decision-making and gradient updates, are a potential solution to this problem. However, only a few comparisons between asynchronous and synchronous RL have been made with physical robots. For this reason, the exact performance benefits of using asynchronous RL methods over synchronous RL methods are still unclear. In this study, we provide a performance comparison between asynchronous and synchronous RL using a physical robotic arm called Franka Emika Panda. Our experiments show that the agents learn faster and attain significantly more returns using asynchronous RL. Our experiments also demonstrate that the learning agent with a faster response time performs better than the agent with a slower response time, even if the agent with a slower response time performs a higher number of gradient updates.
