自己誘導・自己正則化アクタークリティック
GRAC: Self-Guided and Self-Regularized Actor-Critic
ターゲットネットワークを使わずにTD学習の発散を抑える自己正則化手法と、方策勾配にゼロ次最適化を組み合わせてQ値の高い行動を探索する自己誘導型の方策改善を提案し、OpenAI Gymで高い性能を示した。
著者: Lin Shao, Yifan You, Mengyuan Yan, Qingyun Sun, Jeannette Bohg
分類: cs.LG, cs.AI, cs.RO, cs.SY, eess.SY, stat.ML
原文アブストラクト
Deep reinforcement learning (DRL) algorithms have successfully been demonstrated on a range of challenging decision making and control tasks. One dominant component of recent deep reinforcement learning algorithms is the target network which mitigates the divergence when learning the Q function. However, target networks can slow down the learning process due to delayed function updates. Our main contribution in this work is a self-regularized TD-learning method to address divergence without requiring a target network. Additionally, we propose a self-guided policy improvement method by combining policy-gradient with zero-order optimization to search for actions associated with higher Q-values in a broad neighborhood. This makes learning more robust to local noise in the Q function approximation and guides the updates of our actor network. Taken together, these components define GRAC, a novel self-guided and self-regularized actor critic algorithm. We evaluate GRAC on the suite of OpenAI gym tasks, achieving or outperforming state of the art in every environment tested.
