一般化ヒンドサイトによる強化学習の効率的なデータ再利用
Generalized Hindsight for Reinforcement Learning
あるタスクで集めた低報酬データを、その行動に適した別タスクへ再ラベリングする近似逆強化学習手法を提案し、マルチタスクのナビゲーションとマニピュレーションでサンプル効率の向上を示した。
著者: Alexander C. Li, Lerrel Pinto, Pieter Abbeel
分類: cs.LG, cs.AI, cs.NE, cs.RO, stat.ML
原文アブストラクト
One of the key reasons for the high sample complexity in reinforcement learning (RL) is the inability to transfer knowledge from one task to another. In standard multi-task RL settings, low-reward data collected while trying to solve one task provides little to no signal for solving that particular task and is hence effectively wasted. However, we argue that this data, which is uninformative for one task, is likely a rich source of information for other tasks. To leverage this insight and efficiently reuse data, we present Generalized Hindsight: an approximate inverse reinforcement learning technique for relabeling behaviors with the right tasks. Intuitively, given a behavior generated under one task, Generalized Hindsight returns a different task that the behavior is better suited for. Then, the behavior is relabeled with this new task before being used by an off-policy RL optimizer. Compared to standard relabeling techniques, Generalized Hindsight provides a substantially more efficient reuse of samples, which we empirically demonstrate on a suite of multi-task navigation and manipulation tasks. Videos and code can be accessed here: https://sites.google.com/view/generalized-hindsight.
