抽象的デモンストレーションと適応的探索による効率的で安定な多段階スパース報酬強化学習
Abstract Demonstrations and Adaptive Exploration for Efficient and Stable Multi-step Sparse Reward Reinforcement Learning
複雑なタスクをサブタスクに分解し、正しい順序で学習させるとともに、習熟度に応じて探索の確率を適応的に変えることで、スパース報酬の多段階タスクを効率的かつ安定に学習する手法A^2を提案。
著者: Xintong Yang, Ze Ji, Jing Wu, Yu-kun Lai
分類: cs.RO, cs.AI, cs.LG
原文アブストラクト
Although Deep Reinforcement Learning (DRL) has been popular in many disciplines including robotics, state-of-the-art DRL algorithms still struggle to learn long-horizon, multi-step and sparse reward tasks, such as stacking several blocks given only a task-completion reward signal. To improve learning efficiency for such tasks, this paper proposes a DRL exploration technique, termed A^2, which integrates two components inspired by human experiences: Abstract demonstrations and Adaptive exploration. A^2 starts by decomposing a complex task into subtasks, and then provides the correct orders of subtasks to learn. During training, the agent explores the environment adaptively, acting more deterministically for well-mastered subtasks and more stochastically for ill-learnt subtasks. Ablation and comparative experiments are conducted on several grid-world tasks and three robotic manipulation tasks. We demonstrate that A^2 can aid popular DRL algorithms (DQN, DDPG, and SAC) to learn more efficiently and stably in these environments.
