USHER: 後知恵経験再生のための不偏サンプリング
USHER: Unbiased Sampling for Hindsight Experience Replay
後知恵経験再生(HER)が確率的環境で価値関数に偏りを生む問題を、重点サンプリングに基づく漸近的に不偏な手法で解決し、ロボット制御タスクで有効性を示した論文。
著者: Liam Schramm, Yunfu Deng, Edgar Granados, Abdeslam Boularias
分類: cs.LG, cs.AI, cs.RO
原文アブストラクト
Dealing with sparse rewards is a long-standing challenge in reinforcement learning (RL). Hindsight Experience Replay (HER) addresses this problem by reusing failed trajectories for one goal as successful trajectories for another. This allows for both a minimum density of reward and for generalization across multiple goals. However, this strategy is known to result in a biased value function, as the update rule underestimates the likelihood of bad outcomes in a stochastic environment. We propose an asymptotically unbiased importance-sampling-based algorithm to address this problem without sacrificing performance on deterministic environments. We show its effectiveness on a range of robotic systems, including challenging high dimensional stochastic environments.
