目標条件付きオフライン強化学習による階層的計画
Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learning
低レベルの目標条件付きオフラインRL方策と高レベルの目標プランナを組み合わせ、長期タスクを階層的に計画する手法を提案した論文。
著者: Jinning Li, Chen Tang, Masayoshi Tomizuka, Wei Zhan
分類: cs.RO, cs.SY, eess.SY
原文アブストラクト
Offline Reinforcement learning (RL) has shown potent in many safe-critical tasks in robotics where exploration is risky and expensive. However, it still struggles to acquire skills in temporally extended tasks. In this paper, we study the problem of offline RL for temporally extended tasks. We propose a hierarchical planning framework, consisting of a low-level goal-conditioned RL policy and a high-level goal planner. The low-level policy is trained via offline RL. We improve the offline training to deal with out-of-distribution goals by a perturbed goal sampling process. The high-level planner selects intermediate sub-goals by taking advantages of model-based planning methods. It plans over future sub-goal sequences based on the learned value function of the low-level policy. We adopt a Conditional Variational Autoencoder to sample meaningful high-dimensional sub-goal candidates and to solve the high-level long-term strategy optimization problem. We evaluate our proposed method in long-horizon driving and robot navigation tasks. Experiments show that our method outperforms baselines with different hierarchical designs and other regular planners without hierarchy in these complex tasks.
