安全強化学習のための制約付き変分方策最適化
Constrained Variational Policy Optimization for Safe Reinforcement Learning
安全制約付き強化学習を確率的推論の枠組みで捉え、EMアルゴリズムにより凸最適化と教師あり学習に分解して安定に方策を学習する手法を提案した。連続ロボットタスクで制約遵守とサンプル効率の向上を示す。
著者: Zuxin Liu, Zhepeng Cen, Vladislav Isenbaev, Wei Liu, Zhiwei Steven Wu, Bo Li, Ding Zhao
分類: cs.LG, cs.AI, cs.RO
原文アブストラクト
Safe reinforcement learning (RL) aims to learn policies that satisfy certain constraints before deploying them to safety-critical applications. Previous primal-dual style approaches suffer from instability issues and lack optimality guarantees. This paper overcomes the issues from the perspective of probabilistic inference. We introduce a novel Expectation-Maximization approach to naturally incorporate constraints during the policy learning: 1) a provable optimal non-parametric variational distribution could be computed in closed form after a convex optimization (E-step); 2) the policy parameter is improved within the trust region based on the optimal variational distribution (M-step). The proposed algorithm decomposes the safe RL problem into a convex optimization phase and a supervised learning phase, which yields a more stable training performance. A wide range of experiments on continuous robotic tasks shows that the proposed method achieves significantly better constraint satisfaction performance and better sample efficiency than baselines. The code is available at https://github.com/liuzuxin/cvpo-safe-rl.
