動的ミラー降下に基づくモデル予測制御でロボット学習を加速する階層フレームワーク
Dynamic Mirror Descent based Model Predictive Control for Accelerating Robot Learning
モデルベース軌道最適化(DMD-MPC)を内側ループ、モデルフリーRLを外側ループとする階層学習法を提案し、MuJoCoベンチマークや2R脚・四足ロボットの実機で収束と実機学習の高速化を示した。
著者: Utkarsh A. Mishra, Soumya R. Samineni, Prakhar Goel, Chandravaran Kunjeti, Himanshu Lodha, Aman Singh, Aditya Sagi, Shalabh Bhatnagar, Shishir Kolathaya
分類: cs.RO
原文アブストラクト
Recent works in Reinforcement Learning (RL) combine model-free (Mf)-RL algorithms with model-based (Mb)-RL approaches to get the best from both: asymptotic performance of Mf-RL and high sample-efficiency of Mb-RL. Inspired by these works, we propose a hierarchical framework that integrates online learning for the Mb-trajectory optimization with off-policy methods for the Mf-RL. In particular, two loops are proposed, where the Dynamic Mirror Descent based Model Predictive Control (DMD-MPC) is used as the inner loop Mb-RL to obtain an optimal sequence of actions. These actions are in turn used to significantly accelerate the outer loop Mf-RL. We show that our formulation is generic for a broad class of MPC-based policies and objectives, and includes some of the well-known Mb-Mf approaches. We finally introduce a new algorithm: Mirror-Descent Model Predictive RL (M-DeMoRL), which uses Cross-Entropy Method (CEM) with elite fractions for the inner loop. Our experiments show faster convergence of the proposed hierarchical approach on benchmark MuJoCo tasks. We also demonstrate hardware training for trajectory tracking in a 2R leg and hardware transfer for robust walking in a quadruped. We show that the inner-loop Mb-RL significantly decreases the number of training iterations required in the real system, thereby validating the proposed approach.
関連論文
- 潜在想像によるパラメータ化行動空間の学習モデルベースRL
- チャンク単位のモデルベース方策最適化モデルベースRL
- モデルベース強化学習における動力学変化下でのリプレイ保持の特性評価モデルベースRL
- モデルベース強化学習のための償却低ランク適応モデルベースRL
- 変化点対応ワールドモデル:動力学シフトの検出と古いリプレイの忘却による回復モデルベースRL
- RENEW: 選好から世界モデルの学習とモデル悪用の修復を目指してモデルベースRL
