スキルベースのモデルベース強化学習
Skill-based Model-based Reinforcement Learning
スキル空間で長期計画を行うために、スキルダイナミクスモデルとスキルレパートリーを同時に学習するSkiMoを提案し、ナビゲーションとマニピュレーションでサンプル効率を改善した。
著者: Lucy Xiaoyang Shi, Joseph J. Lim, Youngwoon Lee
分類: cs.LG, cs.AI, cs.RO
原文アブストラクト
Model-based reinforcement learning (RL) is a sample-efficient way of learning complex behaviors by leveraging a learned single-step dynamics model to plan actions in imagination. However, planning every action for long-horizon tasks is not practical, akin to a human planning out every muscle movement. Instead, humans efficiently plan with high-level skills to solve complex tasks. From this intuition, we propose a Skill-based Model-based RL framework (SkiMo) that enables planning in the skill space using a skill dynamics model, which directly predicts the skill outcomes, rather than predicting all small details in the intermediate states, step by step. For accurate and efficient long-term planning, we jointly learn the skill dynamics model and a skill repertoire from prior experience. We then harness the learned skill dynamics model to accurately simulate and plan over long horizons in the skill space, which enables efficient downstream learning of long-horizon, sparse reward tasks. Experimental results in navigation and manipulation domains show that SkiMo extends the temporal horizon of model-based approaches and improves the sample efficiency for both model-based RL and skill-based RL. Code and videos are available at https://clvrai.com/skimo
関連論文
- 潜在想像によるパラメータ化行動空間の学習モデルベースRL
- チャンク単位のモデルベース方策最適化モデルベースRL
- モデルベース強化学習における動力学変化下でのリプレイ保持の特性評価モデルベースRL
- モデルベース強化学習のための償却低ランク適応モデルベースRL
- 変化点対応ワールドモデル:動力学シフトの検出と古いリプレイの忘却による回復モデルベースRL
- RENEW: 選好から世界モデルの学習とモデル悪用の修復を目指してモデルベースRL
