実世界ロボット強化学習のための反事実予測のオフライン学習
Offline Learning of Counterfactual Predictions for Real-World Robotic Reinforcement Learning
視覚と力覚のマルチモーダル観測から関節速度を出力する方策を、オフラインサンプルで学習した反事実予測と力覚フィードバックを組み合わせて、終端報酬のみで効率的に実世界強化学習する手法を提案した論文。
著者: Jun Jin, Daniel Graves, Cameron Haigh, Jun Luo, Martin Jagersand
分類: cs.RO, cs.AI
原文アブストラクト
We consider real-world reinforcement learning (RL) of robotic manipulation tasks that involve both visuomotor skills and contact-rich skills. We aim to train a policy that maps multimodal sensory observations (vision and force) to a manipulator's joint velocities under practical considerations. We propose to use offline samples to learn a set of general value functions (GVFs) that make counterfactual predictions from the visual inputs. We show that combining the offline learned counterfactual predictions with force feedbacks in online policy learning allows efficient reinforcement learning given only a terminal (success/failure) reward. We argue that the learned counterfactual predictions form a compact and informative representation that enables sample efficiency and provides auxiliary reward signals that guide online explorations towards contact-rich states. Various experiments in simulation and real-world settings were performed for evaluation. Recordings of the real-world robot training can be found via https://sites.google.com/view/realrl.
