報酬適応型強化学習：二足歩行のための動的方策勾配最適化
Reward-Adaptive Reinforcement Learning: Dynamic Policy Gradient Optimization for Bipedal Locomotion
複数の報酬成分ごとに価値関数を学習し動的重み付けで方策勾配を統合するHDPGを提案し、二足歩行制御で従来の報酬和手法を上回り実機転移も示した。
著者: Changxin Huang, Guangrun Wang, Zhibo Zhou, Ronghui Zhang, Liang Lin
分類: cs.RO
原文アブストラクト
Controlling a non-statically bipedal robot is challenging due to the complex dynamics and multi-criterion optimization involved. Recent works have demonstrated the effectiveness of deep reinforcement learning (DRL) for simulation and physical robots. In these methods, the rewards from different criteria are normally summed to learn a single value function. However, this may cause the loss of dependency information between hybrid rewards and lead to a sub-optimal policy. In this work, we propose a novel reward-adaptive reinforcement learning for biped locomotion, allowing the control policy to be simultaneously optimized by multiple criteria using a dynamic mechanism. The proposed method applies a multi-head critic to learn a separate value function for each reward component. This leads to hybrid policy gradient. We further propose dynamic weight, allowing each component to optimize the policy with different priorities. This hybrid and dynamic policy gradient (HDPG) design makes the agent learn more efficiently. We show that the proposed method outperforms summed-up-reward approaches and is able to transfer to physical robots. The sim-to-real and MuJoCo results further demonstrate the effectiveness and generalization of HDPG.
