Ctrl-Z：強化学習の不安定性からの回復
Ctrl-Z: Recovering from Instability in Reinforcement Learning
学習中のエージェントの履歴を保持し、性能が大きく低下したら過去のパラメータに戻すことで、強化学習の不安定な学習を安定化させる手法を提案した論文。
著者: Vibhavari Dasagi, Jake Bruce, Thierry Peynot, Jürgen Leitner
分類: cs.LG, cs.RO, stat.ML
原文アブストラクト
When learning behavior, training data is often generated by the learner itself; this can result in unstable training dynamics, and this problem has particularly important applications in safety-sensitive real-world control tasks such as robotics. In this work, we propose a principled and model-agnostic approach to mitigate the issue of unstable learning dynamics by maintaining a history of a reinforcement learning agent over the course of training, and reverting to the parameters of a previous agent whenever performance significantly decreases. We develop techniques for evaluating this performance through statistical hypothesis testing of continued improvement, and evaluate them on a standard suite of challenging benchmark tasks involving continuous control of simulated robots. We show improvements over state-of-the-art reinforcement learning algorithms in performance and robustness to hyperparameters, outperforming DDPG in 5 out of 6 evaluation environments and showing no decrease in performance with TD3, which is known to be relatively stable. In this way, our approach takes an important step towards increasing data efficiency and stability in training for real-world robotic applications.
