サンプル効率の高い目標条件付き強化学習のための計量残差ネットワーク
Metric Residual Networks for Sample Efficient Goal-Conditioned Reinforcement Learning
目標条件付き強化学習において、行動価値関数を計量と非対称残差に分解する新しいニューラルネットワークMRNを提案し、サンプル効率を大幅に改善した。
著者: Bo Liu, Yihao Feng, Qiang Liu, Peter Stone
分類: cs.LG, cs.AI, cs.RO
原文アブストラクト
Goal-conditioned reinforcement learning (GCRL) has a wide range of potential real-world applications, including manipulation and navigation problems in robotics. Especially in such robotics tasks, sample efficiency is of the utmost importance for GCRL since, by default, the agent is only rewarded when it reaches its goal. While several methods have been proposed to improve the sample efficiency of GCRL, one relatively under-studied approach is the design of neural architectures to support sample efficiency. In this work, we introduce a novel neural architecture for GCRL that achieves significantly better sample efficiency than the commonly-used monolithic network architecture. The key insight is that the optimal action-value function Q^*(s, a, g) must satisfy the triangle inequality in a specific sense. Furthermore, we introduce the metric residual network (MRN) that deliberately decomposes the action-value function Q(s,a,g) into the negated summation of a metric plus a residual asymmetric component. MRN provably approximates any optimal action-value function Q^*(s,a,g), thus making it a fitting neural architecture for GCRL. We conduct comprehensive experiments across 12 standard benchmark environments in GCRL. The empirical results demonstrate that MRN uniformly outperforms other state-of-the-art GCRL neural architectures in terms of sample efficiency.
