本質的動機付けを備えた目標条件付き方策の学習
Learn Goal-Conditioned Policy with Intrinsic Motivation for Deep Reinforcement Learning
潜在変数で多様な状態を探索する抽象方策と識別器を組み合わせ、識別器を本質的報酬として目標条件付き方策を教師なしで学習するGPIMを提案し、ロボットタスクで有効性を示した。
著者: Jinxin Liu, Donglin Wang, Qiangxing Tian, Zhengyu Chen
分類: cs.LG, cs.RO
原文アブストラクト
It is of significance for an agent to learn a widely applicable and general-purpose policy that can achieve diverse goals including images and text descriptions. Considering such perceptually-specific goals, the frontier of deep reinforcement learning research is to learn a goal-conditioned policy without hand-crafted rewards. To learn this kind of policy, recent works usually take as the reward the non-parametric distance to a given goal in an explicit embedding space. From a different viewpoint, we propose a novel unsupervised learning approach named goal-conditioned policy with intrinsic motivation (GPIM), which jointly learns both an abstract-level policy and a goal-conditioned policy. The abstract-level policy is conditioned on a latent variable to optimize a discriminator and discovers diverse states that are further rendered into perceptually-specific goals for the goal-conditioned policy. The learned discriminator serves as an intrinsic reward function for the goal-conditioned policy to imitate the trajectory induced by the abstract-level policy. Experiments on various robotic tasks demonstrate the effectiveness and efficiency of our proposed GPIM method which substantially outperforms prior techniques.
