まず戻ってから探索せよ：内在的動機付けにおけるポスト探索の利点
First Go, then Post-Explore: the Benefits of Post-Exploration in Intrinsic Motivation
強化学習の探索手法Go-Exploreの「ポスト探索」をIMGEP枠組みで分離検証し、多様な状態到達と性能向上に有効で実装も容易だと示した。
著者: Zhao Yang, Thomas M. Moerland, Mike Preuss, Aske Plaat
分類: cs.LG, cs.AI, cs.RO
原文アブストラクト
Go-Explore achieved breakthrough performance on challenging reinforcement learning (RL) tasks with sparse rewards. The key insight of Go-Explore was that successful exploration requires an agent to first return to an interesting state ('Go'), and only then explore into unknown terrain ('Explore'). We refer to such exploration after a goal is reached as 'post-exploration'. In this paper, we present a clear ablation study of post-exploration in a general intrinsically motivated goal exploration process (IMGEP) framework, that the Go-Explore paper did not show. We study the isolated potential of post-exploration, by turning it on and off within the same algorithm under both tabular and deep RL settings on both discrete navigation and continuous control tasks. Experiments on a range of MiniGrid and Mujoco environments show that post-exploration indeed helps IMGEP agents reach more diverse states and boosts their performance. In short, our work suggests that RL researchers should consider to use post-exploration in IMGEP when possible since it is effective, method-agnostic and easy to implement.
