実演と成功体験を報酬再ラベリングで活用するSACR2
Learning from demonstrations with SACR2: Soft Actor-Critic with Reward Relabeling
Soft Actor-Criticに実演データと成功エピソードへの報酬ボーナス・再ラベリングを組み込み、スパース報酬のロボットマニピュレーション学習を効率化する手法を提案・評価した論文。
著者: Jesus Bujalance Martin, Raphael Chekroun, Fabien Moutarde
分類: cs.LG, cs.RO
原文アブストラクト
During recent years, deep reinforcement learning (DRL) has made successful incursions into complex decision-making applications such as robotics, autonomous driving or video games. Off-policy algorithms tend to be more sample-efficient than their on-policy counterparts, and can additionally benefit from any off-policy data stored in the replay buffer. Expert demonstrations are a popular source for such data: the agent is exposed to successful states and actions early on, which can accelerate the learning process and improve performance. In the past, multiple ideas have been proposed to make good use of the demonstrations in the buffer, such as pretraining on demonstrations only or minimizing additional cost functions. We carry on a study to evaluate several of these ideas in isolation, to see which of them have the most significant impact. We also present a new method for sparse-reward tasks, based on a reward bonus given to demonstrations and successful episodes. First, we give a reward bonus to the transitions coming from demonstrations to encourage the agent to match the demonstrated behaviour. Then, upon collecting a successful episode, we relabel its transitions with the same bonus before adding them to the replay buffer, encouraging the agent to also match its previous successes. The base algorithm for our experiments is the popular Soft Actor-Critic (SAC), a state-of-the-art off-policy algorithm for continuous action spaces. Our experiments focus on manipulation robotics, specifically on a 3D reaching task for a robotic arm in simulation. We show that our method SACR2 based on reward relabeling improves the performance on this task, even in the absence of demonstrations.
関連論文
- 高次行動監督による強力な方策クラス模倣学習/強化学習
- ConTrack: 適応的トレードオフ制御による拘束付き手の動作追跡模倣学習/強化学習
- 不完全なデモンストレーションからの学習:時間行動木ガイドによる軌道修復模倣学習/強化学習
- IN-RIL: 模倣学習と強化学習を交互に組み合わせた方策ファインチューニング模倣学習/強化学習
- DFM: 表現豊かなダンス動作学習のための深層フーリエ模倣模倣学習/強化学習
- BMP:Bスプラインと運動プリミティブの橋渡し模倣学習/強化学習
