安全で効率的な強化学習のための乗算型価値関数
A Multiplicative Value Function for Safe and Efficient Reinforcement Learning
安全性クリティックと報酬クリティックを掛け合わせる新しい価値関数により、制約違反を抑えつつサンプル効率を高める安全強化学習手法を提案し、PPOとSACに統合してロボットナビゲーションを含む4環境で評価した。
著者: Nick Bührer, Zhejun Zhang, Alexander Liniger, Fisher Yu, Luc Van Gool
分類: cs.RO, cs.LG
原文アブストラクト
An emerging field of sequential decision problems is safe Reinforcement Learning (RL), where the objective is to maximize the reward while obeying safety constraints. Being able to handle constraints is essential for deploying RL agents in real-world environments, where constraint violations can harm the agent and the environment. To this end, we propose a safe model-free RL algorithm with a novel multiplicative value function consisting of a safety critic and a reward critic. The safety critic predicts the probability of constraint violation and discounts the reward critic that only estimates constraint-free returns. By splitting responsibilities, we facilitate the learning task leading to increased sample efficiency. We integrate our approach into two popular RL algorithms, Proximal Policy Optimization and Soft Actor-Critic, and evaluate our method in four safety-focused environments, including classical RL benchmarks augmented with safety constraints and robot navigation tasks with images and raw Lidar scans as observations. Finally, we make the zero-shot sim-to-real transfer where a differential drive robot has to navigate through a cluttered room. Our code can be found at https://github.com/nikeke19/Safe-Mult-RL.
