楽観的探索と分布補正を組み合わせたオフポリシー強化学習
Off-policy Reinforcement Learning with Optimistic Exploration and Distribution Correction
不確実性に対する楽観性に基づく探索ポリシーとDICEによる分布補正を組み合わせ、オフポリシー強化学習のサンプル効率を改善した研究。
著者: Jiachen Li, Shuo Cheng, Zhenyu Liao, Huayan Wang, William Yang Wang, Qinxun Bai
分類: cs.LG, cs.RO
原文アブストラクト
Improving the sample efficiency of reinforcement learning algorithms requires effective exploration. Following the principle of $\textit{optimism in the face of uncertainty}$ (OFU), we train a separate exploration policy to maximize the approximate upper confidence bound of the critics in an off-policy actor-critic framework. However, this introduces extra differences between the replay buffer and the target policy regarding their stationary state-action distributions. To mitigate the off-policy-ness, we adapt the recently introduced DICE framework to learn a distribution correction ratio for off-policy RL training. In particular, we correct the training distribution for both policies and critics. Empirically, we evaluate our proposed method in several challenging continuous control tasks and show superior performance compared to state-of-the-art methods. We also conduct extensive ablation studies to demonstrate the effectiveness and rationality of the proposed method.
