時間的一貫性のみに基づく簡素化強化学習
Simplified Temporal Consistency Reinforcement Learning
潜在時間的一貫性で学習した潜在ダイナミクスモデルのみを用いる簡素な表現学習が、高次元の運動タスクで高性能な強化学習を実現し、既存手法より高速に学習できることを示した。
著者: Yi Zhao, Wenshuai Zhao, Rinu Boney, Juho Kannala, Joni Pajarinen
分類: cs.LG, cs.RO
原文アブストラクト
Reinforcement learning is able to solve complex sequential decision-making tasks but is currently limited by sample efficiency and required computation. To improve sample efficiency, recent work focuses on model-based RL which interleaves model learning with planning. Recent methods further utilize policy learning, value estimation, and, self-supervised learning as auxiliary objectives. In this paper we show that, surprisingly, a simple representation learning approach relying only on a latent dynamics model trained by latent temporal consistency is sufficient for high-performance RL. This applies when using pure planning with a dynamics model conditioned on the representation, but, also when utilizing the representation as policy and value function features in model-free RL. In experiments, our approach learns an accurate dynamics model to solve challenging high-dimensional locomotion tasks with online planners while being 4.1 times faster to train compared to ensemble-based methods. With model-free RL without planning, especially on high-dimensional tasks, such as the DeepMind Control Suite Humanoid and Dog tasks, our approach outperforms model-free methods by a large margin and matches model-based methods' sample efficiency while training 2.4 times faster.
関連論文
- 潜在想像によるパラメータ化行動空間の学習モデルベースRL
- チャンク単位のモデルベース方策最適化モデルベースRL
- モデルベース強化学習における動力学変化下でのリプレイ保持の特性評価モデルベースRL
- モデルベース強化学習のための償却低ランク適応モデルベースRL
- 変化点対応ワールドモデル:動力学シフトの検出と古いリプレイの忘却による回復モデルベースRL
- RENEW: 選好から世界モデルの学習とモデル悪用の修復を目指してモデルベースRL
