相対ピアソンダイバージェンスを用いた近接方策最適化
Proximal Policy Optimization with Relative Pearson Divergence
PPOの正則化目標を相対ピアソンダイバージェンスで明確化し、密度比の非対称性に整合する閾値設計を導出したPPO-RPEを提案。4つのベンチマークで従来法と同等以上の性能を示した。
分類: cs.LG, cs.RO
原文アブストラクト
The recent remarkable progress of deep reinforcement learning (DRL) stands on regularization of policy for stable and efficient learning. A popular method, named proximal policy optimization (PPO), has been introduced for this purpose. PPO clips density ratio of the latest and baseline policies with a threshold, while its minimization target is unclear. As another problem of PPO, the symmetric threshold is given numerically while the density ratio itself is in asymmetric domain, thereby causing unbalanced regularization of the policy. This paper therefore proposes a new variant of PPO by considering a regularization problem of relative Pearson (RPE) divergence, so-called PPO-RPE. This regularization yields the clear minimization target, which constrains the latest policy to the baseline one. Through its analysis, the intuitive threshold-based design consistent with the asymmetry of the threshold and the domain of density ratio can be derived. Through four benchmark tasks, PPO-RPE performed as well as or better than the conventional methods in terms of the task performance by the learned policy.
