TRC: 安全強化学習のための信頼領域条件付きバリュー・アット・リスク
TRC: Trust Region Conditional Value at Risk for Safe Reinforcement Learning
CVaR制約付きの安全強化学習手法TRCを提案し、信頼領域内でCVaRの上界を微分可能に近似して方策勾配を求める。シミュレーションと実機で制約を満たしつつ性能を1.93倍向上させた。
著者: Dohyeong Kim, Songhwai Oh
分類: cs.RO, cs.LG
原文アブストラクト
As safety is of paramount importance in robotics, reinforcement learning that reflects safety, called safe RL, has been studied extensively. In safe RL, we aim to find a policy which maximizes the desired return while satisfying the defined safety constraints. There are various types of constraints, among which constraints on conditional value at risk (CVaR) effectively lower the probability of failures caused by high costs since CVaR is a conditional expectation obtained above a certain percentile. In this paper, we propose a trust region-based safe RL method with CVaR constraints, called TRC. We first derive the upper bound on CVaR and then approximate the upper bound in a differentiable form in a trust region. Using this approximation, a subproblem to get policy gradients is formulated, and policies are trained by iteratively solving the subproblem. TRC is evaluated through safe navigation tasks in simulations with various robots and a sim-to-real environment with a Jackal robot from Clearpath. Compared to other safe RL methods, the performance is improved by 1.93 times while the constraints are satisfied in all experiments.
