脚式ロコモーションのための制約付き強化学習アルゴリズムの評価
Evaluation of Constrained Reinforcement Learning Algorithms for Legged Locomotion
脚式ロボットの制御に制約付きマルコフ決定過程(CMDP)を適用し、複数の制約付き方策最適化アルゴリズムを比較して、物理制約を考慮したsim-to-real転移の有効性を示した。
著者: Joonho Lee, Lukas Schroth, Victor Klemm, Marko Bjelonic, Alexander Reske, Marco Hutter
分類: cs.RO, cs.SY, eess.SY
原文アブストラクト
Shifting from traditional control strategies to Deep Reinforcement Learning (RL) for legged robots poses inherent challenges, especially when addressing real-world physical constraints during training. While high-fidelity simulations provide significant benefits, they often bypass these essential physical limitations. In this paper, we experiment with the Constrained Markov Decision Process (CMDP) framework instead of the conventional unconstrained RL for robotic applications. We perform a comparative study of different constrained policy optimization algorithms to identify suitable methods for practical implementation. Our robot experiments demonstrate the critical role of incorporating physical constraints, yielding successful sim-to-real transfers, and reducing operational errors on physical systems. The CMDP formulation streamlines the training process by separately handling constraints from rewards. Our findings underscore the potential of constrained RL for the effective development and deployment of learned controllers in robotics.
