強化学習によるロボット制御における行動平滑性のための勾配ベース正則化
Gradient-based Regularization for Action Smoothness in Robotic Control with Reinforcement Learning
強化学習でロボットを制御する際に生じるギザギザした行動を抑えるため、行動の勾配差を減らすGrad-CAPSを提案し、複数のタスクで性能と平滑性の両立を確認した。
著者: I Lee, Hoang-Giang Cao, Cong-Tinh Dao, Yu-Cheng Chen, I-Chen Wu
分類: cs.RO
原文アブストラクト
Deep Reinforcement Learning (DRL) has achieved remarkable success, ranging from complex computer games to real-world applications, showing the potential for intelligent agents capable of learning in dynamic environments. However, its application in real-world scenarios presents challenges, including the jerky problem, in which jerky trajectories not only compromise system safety but also increase power consumption and shorten the service life of robotic and autonomous systems. To address jerky actions, a method called conditioning for action policy smoothness (CAPS) was proposed by adding regularization terms to reduce the action changes. This paper further proposes a novel method, named Gradient-based CAPS (Grad-CAPS), that modifies CAPS by reducing the difference in the gradient of action and then uses displacement normalization to enable the agent to adapt to invariant action scales. Consequently, our method effectively reduces zigzagging action sequences while enhancing policy expressiveness and the adaptability of our method across diverse scenarios and environments. In the experiments, we integrated Grad-CAPS with different reinforcement learning algorithms and evaluated its performance on various robotic-related tasks in DeepMind Control Suite and OpenAI Gym environments. The results demonstrate that Grad-CAPS effectively improves performance while maintaining a comparable level of smoothness compared to CAPS and Vanilla agents.
