Critic PI2:経路積分による方策改善と深層アクター・クリティック強化学習を組み合わせた連続計画の習得
Critic PI2: Master Continuous Planning via Policy Improvement with Path Integrals and Deep Actor-Critic Reinforcement Learning
軌道最適化・深層アクタークリティック学習・モデルベース強化学習を統合し、連続行動空間での計画を可能にするCritic PI2を提案。倒立振子実験でサンプル効率とリアルタイム性の向上を示した。
著者: Jiajun Fan, He Ba, Xian Guo, Jianye Hao
分類: cs.LG, cs.NE, cs.RO
原文アブストラクト
Constructing agents with planning capabilities has long been one of the main challenges in the pursuit of artificial intelligence. Tree-based planning methods from AlphaGo to Muzero have enjoyed huge success in discrete domains, such as chess and Go. Unfortunately, in real-world applications like robot control and inverted pendulum, whose action space is normally continuous, those tree-based planning techniques will be struggling. To address those limitations, in this paper, we present a novel model-based reinforcement learning frameworks called Critic PI2, which combines the benefits from trajectory optimization, deep actor-critic learning, and model-based reinforcement learning. Our method is evaluated for inverted pendulum models with applicability to many continuous control systems. Extensive experiments demonstrate that Critic PI2 achieved a new state of the art in a range of challenging continuous domains. Furthermore, we show that planning with a critic significantly increases the sample efficiency and real-time performance. Our work opens a new direction toward learning the components of a model-based planning system and how to use them.
関連論文
- テスト時モデルベース強化学習による世界モデルへの記憶の符号化モデルベース強化学習
- 再考:世界モデルにおける比較的想像を通じた探索と計画モデルベース強化学習
- アクションジェット:行動条件付きデータ多様体の局所形状の学習モデルベース強化学習
- 未モデル状態依存性下での学習クリティックによる最適制御モデルベース強化学習
- VIGOR: モデルベース強化学習における潜在空間一貫性によるゼロショット視覚汎化モデルベース強化学習
- 行動なし時系列からの動的埋め込みによる転移可能な方策学習モデルベース強化学習
