確率的完全探索を用いた強化学習
Reinforcement Learning with Probabilistically Complete Exploration
RRTなどの計画アルゴリズムで探索を初期解探索問題として解き、その解をデモとして方策を初期化してから強化学習で洗練させるR3Lを提案し、サンプル効率と安定性を向上させた。
著者: Philippe Morere, Gilad Francis, Tom Blau, Fabio Ramos
分類: cs.LG, cs.RO, stat.ML
原文アブストラクト
Balancing exploration and exploitation remains a key challenge in reinforcement learning (RL). State-of-the-art RL algorithms suffer from high sample complexity, particularly in the sparse reward case, where they can do no better than to explore in all directions until the first positive rewards are found. To mitigate this, we propose Rapidly Randomly-exploring Reinforcement Learning (R3L). We formulate exploration as a search problem and leverage widely-used planning algorithms such as Rapidly-exploring Random Tree (RRT) to find initial solutions. These solutions are used as demonstrations to initialize a policy, then refined by a generic RL algorithm, leading to faster and more stable convergence. We provide theoretical guarantees of R3L exploration finding successful solutions, as well as bounds for its sampling complexity. We experimentally demonstrate the method outperforms classic and intrinsic exploration techniques, requiring only a fraction of exploration samples and achieving better asymptotic performance.
