スキルベース強化学習のための内在報酬マッチング
Skill-Based Reinforcement Learning with Intrinsic Reward Matching
事前学習したスキル識別器を活用し、内在報酬とタスク報酬をマッチングさせることで、環境サンプルなしに最適なスキルを選択し、サンプル効率よくファインチューニングする手法を提案。
著者: Ademi Adeniji, Amber Xie, Pieter Abbeel
分類: cs.LG, cs.AI, cs.RO
原文アブストラクト
While unsupervised skill discovery has shown promise in autonomously acquiring behavioral primitives, there is still a large methodological disconnect between task-agnostic skill pretraining and downstream, task-aware finetuning. We present Intrinsic Reward Matching (IRM), which unifies these two phases of learning via the $\textit{skill discriminator}$, a pretraining model component often discarded during finetuning. Conventional approaches finetune pretrained agents directly at the policy level, often relying on expensive environment rollouts to empirically determine the optimal skill. However, often the most concise yet complete description of a task is the reward function itself, and skill learning methods learn an $\textit{intrinsic}$ reward function via the discriminator that corresponds to the skill policy. We propose to leverage the skill discriminator to $\textit{match}$ the intrinsic and downstream task rewards and determine the optimal skill for an unseen task without environment samples, consequently finetuning with greater sample-efficiency. Furthermore, we generalize IRM to sequence skills for complex, long-horizon tasks and demonstrate that IRM enables us to utilize pretrained skills far more effectively than previous skill selection methods on both the Fetch tabletop and Franka Kitchen robot manipulation benchmarks.
